From d31f74ec5e88aa3a911f7c44eb2e9002a6176094 Mon Sep 17 00:00:00 2001 From: Kyle Squizzato Date: Thu, 27 Aug 2026 13:54:52 -0700 Subject: [PATCH 01/14] feat: Support OpenShift targets for development Add openshift-* targets for supporting an OpenShift development workflow similar to how kind-* Makefile targets work. This will allow users to deploy onto a target OpenShift cluster. Relates to HYPERSHELL-44 Assisted-by: Cursor Grok 4.6 --- .gitignore | 6 + DEVELOPMENT.md | 103 +- Makefile | 113 +- deploy/hub/kustomization.yaml | 1 - deploy/openshift/keycloak-networkpolicy.yaml | 26 + deploy/openshift/keycloak-route.yaml | 14 + deploy/openshift/kustomization.yaml | 2 + deploy/openshift/route.yaml | 15 + deploy/openshift/scc.yaml | 11 +- scripts/cluster/down.sh | 8 + scripts/cluster/drivers/kind.sh | 31 + scripts/cluster/drivers/openshift.sh | 1287 ++++++++++++++++++ scripts/cluster/lib.sh | 255 ++++ scripts/cluster/lib_test.sh | 464 +++++++ scripts/cluster/rewrite-namespaces.py | 214 +++ scripts/cluster/status.sh | 8 + scripts/cluster/swap.sh | 24 + scripts/cluster/teardown.sh | 8 + scripts/cluster/up.sh | 8 + skills/RECONCILE.md | 35 +- specs/platform/e2e-testing.spec.md | 56 +- specs/platform/local-development.spec.md | 19 +- specs/platform/openshift-development.spec.md | 152 ++- 23 files changed, 2755 insertions(+), 105 deletions(-) create mode 100644 deploy/openshift/keycloak-networkpolicy.yaml create mode 100644 deploy/openshift/keycloak-route.yaml create mode 100755 scripts/cluster/down.sh create mode 100755 scripts/cluster/drivers/kind.sh create mode 100755 scripts/cluster/drivers/openshift.sh create mode 100755 scripts/cluster/lib.sh create mode 100755 scripts/cluster/lib_test.sh create mode 100755 scripts/cluster/rewrite-namespaces.py create mode 100755 scripts/cluster/status.sh create mode 100755 scripts/cluster/swap.sh create mode 100755 scripts/cluster/teardown.sh create mode 100755 scripts/cluster/up.sh diff --git a/.gitignore b/.gitignore index d42cc663..0f549bb0 100644 --- a/.gitignore +++ b/.gitignore @@ -36,6 +36,12 @@ apm_modules/ # Kind swap tracking .kind-swaps +# OpenShift per-namespace swap tracking +.openshift-swaps/ + +# Temporary kustomize wrap dirs used by openshift-up +.openshift-render.*/ + # Local tool binaries (built by make kind-prereqs) bin/ diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md index 0f1846bf..42d045d3 100644 --- a/DEVELOPMENT.md +++ b/DEVELOPMENT.md @@ -1,10 +1,12 @@ # Local Development Environment -HyperShell provides a single-command local development environment using -[Kind](https://kind.sigs.k8s.io/) (Kubernetes in Docker) clusters. The -environment deploys all platform components -- API server, control plane, and -web console -- so developers can test changes end-to-end without external -infrastructure. +HyperShell provides a single-command local development environment. Kind +([Kind](https://kind.sigs.k8s.io/), Kubernetes in Docker) is the default +single-tenant path. OpenShift uses the same make-target pattern +(`make openshift-up`, `make openshift--up`) against an existing +cluster and an ephemeral namespace group. The environment deploys all platform +components -- API server, control plane, and web console -- so developers can +test changes end-to-end. ## Prerequisites @@ -284,6 +286,83 @@ make kind-status # Show cluster info, pods, services, swap state `make kind-up` is idempotent -- running it again on an existing cluster reapplies manifests and waits for readiness. Swapped components are preserved. +## OpenShift Development + +`make openshift-up` deploys the same stack into an isolated namespace group on +an OpenShift cluster selected by the current kubeconfig context. It does not +create the cluster. An administrator must already have provisioned the shared +Gateway, GatewayClass, certificate issuer, and wildcard certificate (see +`deploy/openshift/infrastructure/GATEWAY-SETUP.md`). + +The platform namespace is the current oc project. Select it first, then bring +the environment up: + +```bash +oc project alice +make openshift-up +``` + +`OPENSHIFT_NAMESPACE` overrides that project when you need to target a +namespace other than the one `oc project -q` reports: + +```bash +OPENSHIFT_NAMESPACE=alice make openshift-up +``` + +The name must be a valid RFC 1123 DNS label of at most 54 characters so the +companion Keycloak namespace `${name}-keycloak` stays within the 63-character +limit. If no project is selected and `OPENSHIFT_NAMESPACE` is unset, the +command stops with an error. + +`make openshift-up` deploys into the project you selected. It does not ask +for confirmation, and it does not require permission to label the namespace. +When the account can patch namespaces, the scripts stamp HyperShell ownership +labels. When it cannot, the scripts warn and continue. Namespaces that already +belong to a different HyperShell environment, and reserved names (`default`, +`kube-*`, `openshift-*`), are still refused. + +The companion Keycloak project `${name}-keycloak` is created with +`oc new-project` when it does not exist (developers can ProjectRequest; they +typically cannot `oc create namespace`). The scripts switch to that project to +apply Keycloak, then switch back to the platform project for the rest of the +stack. OpenShift's default project NetworkPolicies only allow ingress from the +same namespace and from `openshift-ingress`, so the overlay also applies +`keycloak-allow-platform` in the Keycloak project. That policy lets the API +server load JWKS and the control plane call the Admin API over the in-cluster +Service. + +This renders `kustomize build deploy/openshift/`, maps `hypershell-system` to +that platform namespace and `keycloak` to `${platform}-keycloak`, applies the +manifests (with prune scoped to this environment), registers the web-console +Route as the Keycloak `hypershell-frontend` redirect URI, seeds a +ManagedCluster, GatewayRelease, ManagedDatabase, and Gateway from this machine +against the API and Keycloak Routes (the API server image has no `curl`), and +prints the API, web-console, and Keycloak Routes. The gateway base domain is +read from the shared Gateway's listener hostname, not from +`GATEWAY_API_BASE_DOMAIN`. + +`make openshift-down` and `make openshift-teardown` are the same command. +There is no OpenShift cluster to destroy. Both delete the platform project +and the companion `${name}-keycloak` project. If project deletion is +forbidden, they delete HyperShell resources inside both projects (including +unlabeled Keycloak) and leave the projects. Labels are not required. + +### Per-component swap + +```bash +make openshift-api-server-up +make openshift-control-plane-up +make openshift-web-console-up +``` + +Each swap builds from the working tree, pushes an immutable image (commit + +namespace) to the OpenShift internal registry, and rolls out that exact +identity. Matching `-down` targets revert to the baseline registry image. +`make openshift-status` reports which components run a working-tree build +and the exact image each one uses. Swap state is tracked per namespace in +`.openshift-swaps/` (gitignored). A subsequent `make openshift-up` preserves +active swaps. + ## Environment Variable Reference | Variable | Default | Description | @@ -306,6 +385,10 @@ reapplies manifests and waits for readiness. Swapped components are preserved. | `KIND_DB_IMAGE` | `registry.access.redhat.com/hi/postgresql:18.4@sha256:9b19...` | Database image for Gateway; override for OSS dev | | `KIND_NO_SUDO` | (unset) | Set to `true` to skip sudo operations | | `KIND_DNS_PORT` | `5553` | Host port for CoreDNS container | +| `OPENSHIFT_NAMESPACE` | `oc project -q` | Override for the platform namespace. Unset, the current oc project is used. Max 54 chars; Keycloak lands in `${name}-keycloak`. | +| `GATEWAY_API_GATEWAY_NAME` | `openshell-grpc-gateway` | Pre-existing shared Gateway name | +| `GATEWAY_API_GATEWAY_NAMESPACE` | `openshift-ingress` | Namespace of the shared Gateway | +| `OPENSHIFT_IMAGE_REGISTRY` | `oc registry info` | Registry used to push swapped images | ## Make Targets @@ -322,6 +405,16 @@ reapplies manifests and waits for readiness. Swapped components are preserved. | `make kind-web-console-up` | Hot reload (default) or build + swap web console | | `make kind-web-console-down` | Revert web console to baseline image | | `make kind-fix-ports` | Re-establish host port forwarding (443) | +| `make openshift-up` | Deploy into an ephemeral OpenShift namespace group | +| `make openshift-down` | Remove the namespace group (platform project and `${name}-keycloak`) | +| `make openshift-teardown` | Same as `openshift-down` (OpenShift has no cluster to destroy) | +| `make openshift-status` | Show namespaces, pods, Routes, Gateway, swap state | +| `make openshift-api-server-up` | Build, push, and swap API server from working tree | +| `make openshift-api-server-down` | Revert API server to baseline image | +| `make openshift-control-plane-up` | Build, push, and swap control plane from working tree | +| `make openshift-control-plane-down` | Revert control plane to baseline image | +| `make openshift-web-console-up` | Build, push, and swap web console from working tree | +| `make openshift-web-console-down` | Revert web console to baseline image | ## Gateway Access diff --git a/Makefile b/Makefile index 4c36a296..76c67eff 100644 --- a/Makefile +++ b/Makefile @@ -66,6 +66,13 @@ HYPERSHELL_DATABASE_IMAGE?= KIND_CONFIG=deploy/kind/kind-config.yaml KIND_DNS_PORT?=5553 +# OpenShift ephemeral-namespace development. The cluster is a precondition; +# these names select the shared Gateway the administrator already provisioned. +# The gateway base domain is discovered from that Gateway's listener hostname. +GATEWAY_API_GATEWAY_NAME?=openshell-grpc-gateway +GATEWAY_API_GATEWAY_NAMESPACE?=openshift-ingress +GATEWAY_IMAGE?=ghcr.io/nvidia/openshell/gateway:0.0.109 + # Service hostnames (routed through the networking Gateway) API_HOSTNAME=api.hypershell.localhost CONSOLE_HOSTNAME=console.hypershell.localhost @@ -84,26 +91,32 @@ help: @echo " HyperShell Makefile" @echo " ===================" @echo "" - @echo " Local Development (Kind)" - @echo " All targets operate on KIND_NAMESPACE (default: hypershell-system)." + @echo " Local Development" + @echo " Targets are kind- or openshift-. They do the same work." + @echo "" + @echo " Kind uses KIND_NAMESPACE (default hypershell-system); kind-up creates the cluster." + @echo " OpenShift uses the current oc project (oc project -q); OPENSHIFT_NAMESPACE overrides." + @echo " OpenShift requires an existing cluster — openshift-up does not create one." + @echo "" + @echo " -up Deploy the stack (Kind also creates the cluster)" + @echo " -down Remove the environment namespace(s)" + @echo " -teardown Kind: destroy the cluster. OpenShift: same as -down" + @echo " -status Show cluster, pods, services/Routes, swap state" + @echo " -api-server-up Build + swap API server from working tree" + @echo " -api-server-down Revert API server to baseline image" + @echo " -control-plane-up Build + swap control plane from working tree" + @echo " -control-plane-down Revert control plane to baseline image" + @echo " -web-console-up Swap web console (Kind: hot reload by default)" + @echo " -web-console-down Revert web console to baseline image" @echo "" + @echo " Kind Specific" @echo " kind-env Print environment variables for local setup" - @echo " kind-up Create cluster + deploy all components (OIDC enabled)" - @echo " LOCAL_IMAGES=true: build from working tree (default)" - @echo " LOCAL_IMAGES=true BUILD_SOURCE=baseline: build from origin/main" - @echo " KIND_SKIP_SEED=true: defer seeding (run kind-seed later)" @echo " kind-seed Seed platform resources into a running cluster" - @echo " kind-down Remove namespace and its resources" - @echo " kind-teardown Destroy Kind cluster, stop cloud-provider-kind" - @echo " kind-status Show cluster info, pods, services, swap state" + @echo " KIND_SKIP_SEED=true: defer seeding during kind-up" @echo " kind-fix-ports Re-establish host port forwarding (443 + 8080)" - @echo " kind-api-server-up Build + swap API server from working tree" - @echo " kind-api-server-down Revert API server to baseline image" - @echo " kind-control-plane-up Build + swap control plane from working tree" - @echo " kind-control-plane-down Revert control plane to baseline image" - @echo " kind-web-console-up Hot reload (default) or build + swap web console (KIND_HOT_RELOAD=false)" - @echo " kind-web-console-down Revert web console to baseline image" @echo " kind-gateway-trust Print SSL_CERT_FILE export so the openshell CLI trusts the dev CA" + @echo " LOCAL_IMAGES=true Build baseline images from the working tree (kind-up)" + @echo " BUILD_SOURCE=baseline With LOCAL_IMAGES=true, build from origin/main" @echo "" @echo " Build" @echo " build-all Build all container images" @@ -305,6 +318,8 @@ export api_server_local control_plane_local web_console_local export build_version build_time export API_HOSTNAME CONSOLE_HOSTNAME HEALTH_HOSTNAME KEYCLOAK_HOSTNAME METRICS_HOSTNAME KEYCLOAK_OIDC_ISSUER export KIND_DNS_PORT +export OPENSHIFT_NAMESPACE OPENSHIFT_IMAGE_REGISTRY +export GATEWAY_API_GATEWAY_NAME GATEWAY_API_GATEWAY_NAMESPACE GATEWAY_IMAGE # Build cloud-provider-kind from a fork that adds BackendTLSPolicy support # (TLS re-encryption to backends). The fork also bundles the podman 6+ kind @@ -386,7 +401,7 @@ kind-env: .PHONY: kind-up kind-up: - @scripts/kind/up.sh + @CLUSTER_DRIVER=kind scripts/cluster/up.sh .PHONY: kind-seed kind-seed: @@ -394,15 +409,15 @@ kind-seed: .PHONY: kind-down kind-down: - @scripts/kind/down.sh + @CLUSTER_DRIVER=kind scripts/cluster/down.sh .PHONY: kind-teardown kind-teardown: - @scripts/kind/teardown.sh + @CLUSTER_DRIVER=kind scripts/cluster/teardown.sh .PHONY: kind-status kind-status: - @scripts/kind/status.sh + @CLUSTER_DRIVER=kind scripts/cluster/status.sh .PHONY: kind-fix-ports kind-fix-ports: @@ -410,32 +425,80 @@ kind-fix-ports: .PHONY: kind-api-server-up kind-api-server-up: - @scripts/kind/swap-component.sh up api-server + @CLUSTER_DRIVER=kind scripts/cluster/swap.sh up api-server .PHONY: kind-api-server-down kind-api-server-down: - @scripts/kind/swap-component.sh down api-server + @CLUSTER_DRIVER=kind scripts/cluster/swap.sh down api-server .PHONY: kind-control-plane-up kind-control-plane-up: - @scripts/kind/swap-component.sh up control-plane + @CLUSTER_DRIVER=kind scripts/cluster/swap.sh up control-plane .PHONY: kind-control-plane-down kind-control-plane-down: - @scripts/kind/swap-component.sh down control-plane + @CLUSTER_DRIVER=kind scripts/cluster/swap.sh down control-plane .PHONY: kind-web-console-up kind-web-console-up: - @scripts/kind/swap-component.sh up web-console + @CLUSTER_DRIVER=kind scripts/cluster/swap.sh up web-console .PHONY: kind-web-console-down kind-web-console-down: - @scripts/kind/swap-component.sh down web-console + @CLUSTER_DRIVER=kind scripts/cluster/swap.sh down web-console .PHONY: kind-gateway-trust kind-gateway-trust: @scripts/kind/gateway-trust.sh +# ============================================================================ +# OpenShift cluster lifecycle - shell logic lives in scripts/cluster/ +# ============================================================================ + +.PHONY: openshift-up +openshift-up: + @CLUSTER_DRIVER=openshift scripts/cluster/up.sh + +.PHONY: openshift-down +openshift-down: + @CLUSTER_DRIVER=openshift scripts/cluster/down.sh + +.PHONY: openshift-teardown +openshift-teardown: + @CLUSTER_DRIVER=openshift scripts/cluster/teardown.sh + +.PHONY: openshift-status +openshift-status: + @CLUSTER_DRIVER=openshift scripts/cluster/status.sh + +.PHONY: openshift-api-server-up +openshift-api-server-up: + @CLUSTER_DRIVER=openshift scripts/cluster/swap.sh up api-server + +.PHONY: openshift-api-server-down +openshift-api-server-down: + @CLUSTER_DRIVER=openshift scripts/cluster/swap.sh down api-server + +.PHONY: openshift-control-plane-up +openshift-control-plane-up: + @CLUSTER_DRIVER=openshift scripts/cluster/swap.sh up control-plane + +.PHONY: openshift-control-plane-down +openshift-control-plane-down: + @CLUSTER_DRIVER=openshift scripts/cluster/swap.sh down control-plane + +.PHONY: openshift-web-console-up +openshift-web-console-up: + @CLUSTER_DRIVER=openshift scripts/cluster/swap.sh up web-console + +.PHONY: openshift-web-console-down +openshift-web-console-down: + @CLUSTER_DRIVER=openshift scripts/cluster/swap.sh down web-console + +.PHONY: openshift-test +openshift-test: + @bash scripts/cluster/lib_test.sh + generate-cli: cd scripts/cli-generator && go run . \ --spec ../../components/api-server/openapi/openapi.yaml \ diff --git a/deploy/hub/kustomization.yaml b/deploy/hub/kustomization.yaml index b404ea71..b19c10f4 100644 --- a/deploy/hub/kustomization.yaml +++ b/deploy/hub/kustomization.yaml @@ -10,4 +10,3 @@ resources: - ../base - route.yaml - scc.yaml - diff --git a/deploy/openshift/keycloak-networkpolicy.yaml b/deploy/openshift/keycloak-networkpolicy.yaml new file mode 100644 index 00000000..3b5d4725 --- /dev/null +++ b/deploy/openshift/keycloak-networkpolicy.yaml @@ -0,0 +1,26 @@ +# oc new-project installs default-deny Ingress policies (same-namespace + +# openshift-ingress only). Platform pods live in a different namespace, so +# without this policy the API server cannot load JWKS and the control plane +# cannot reach the Keycloak Admin API. +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: keycloak-allow-platform + namespace: keycloak + labels: + app.kubernetes.io/name: hypershell + app.kubernetes.io/component: network-policy +spec: + podSelector: + matchLabels: + app: keycloak + policyTypes: + - Ingress + ingress: + - from: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: hypershell-system + ports: + - protocol: TCP + port: 8080 diff --git a/deploy/openshift/keycloak-route.yaml b/deploy/openshift/keycloak-route.yaml new file mode 100644 index 00000000..63b9a88f --- /dev/null +++ b/deploy/openshift/keycloak-route.yaml @@ -0,0 +1,14 @@ +apiVersion: route.openshift.io/v1 +kind: Route +metadata: + name: keycloak + namespace: keycloak +spec: + to: + kind: Service + name: keycloak-service + port: + targetPort: http + tls: + termination: edge + insecureEdgeTerminationPolicy: Redirect diff --git a/deploy/openshift/kustomization.yaml b/deploy/openshift/kustomization.yaml index 02a4d510..578f1798 100644 --- a/deploy/openshift/kustomization.yaml +++ b/deploy/openshift/kustomization.yaml @@ -7,6 +7,8 @@ resources: - certificates.yaml - networkpolicies.yaml - route.yaml + - keycloak-route.yaml + - keycloak-networkpolicy.yaml - scc.yaml patches: - patch: | diff --git a/deploy/openshift/route.yaml b/deploy/openshift/route.yaml index ec3d6525..84f96100 100644 --- a/deploy/openshift/route.yaml +++ b/deploy/openshift/route.yaml @@ -12,3 +12,18 @@ spec: tls: termination: edge insecureEdgeTerminationPolicy: Redirect +--- +apiVersion: route.openshift.io/v1 +kind: Route +metadata: + name: hypershell-web-console + namespace: hypershell-system +spec: + to: + kind: Service + name: hypershell-web-console + port: + targetPort: http + tls: + termination: edge + insecureEdgeTerminationPolicy: Redirect diff --git a/deploy/openshift/scc.yaml b/deploy/openshift/scc.yaml index c344898b..25d66b6e 100644 --- a/deploy/openshift/scc.yaml +++ b/deploy/openshift/scc.yaml @@ -1,7 +1,11 @@ +# SCC *use* is namespace-scoped: a RoleBinding to a built-in SCC ClusterRole +# grants use within one namespace. Ephemeral environments cannot share a +# ClusterRoleBinding name, and namespace-scoped access is enough for use. apiVersion: rbac.authorization.k8s.io/v1 -kind: ClusterRoleBinding +kind: RoleBinding metadata: name: hypershell-controller-scc + namespace: hypershell-system roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole @@ -13,10 +17,13 @@ subjects: --- # Sandbox pods require the privileged SCC for container-in-container # execution (see specs/platform/openshell-gateway.spec.md §OpenShift). +# The controller creates additional per-tenant RoleBindings at runtime +# (reconcileOpenShiftSCC) once a privileged actor has granted it bind. apiVersion: rbac.authorization.k8s.io/v1 -kind: ClusterRoleBinding +kind: RoleBinding metadata: name: hypershell-sandbox-scc + namespace: hypershell-system roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole diff --git a/scripts/cluster/down.sh b/scripts/cluster/down.sh new file mode 100755 index 00000000..94e6fc6f --- /dev/null +++ b/scripts/cluster/down.sh @@ -0,0 +1,8 @@ +#!/usr/bin/env bash +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# shellcheck source=lib.sh +source "${SCRIPT_DIR}/lib.sh" +load_cluster_driver +cluster_down diff --git a/scripts/cluster/drivers/kind.sh b/scripts/cluster/drivers/kind.sh new file mode 100755 index 00000000..6e60ff8a --- /dev/null +++ b/scripts/cluster/drivers/kind.sh @@ -0,0 +1,31 @@ +#!/usr/bin/env bash +# Kind lifecycle driver. Wraps the existing scripts/kind/ entrypoints so +# make kind-* keeps today's behavior and a new infrastructure target only +# has to add a driver file. +set -euo pipefail + +KIND_SCRIPTS="${REPO_ROOT}/scripts/kind" + +cluster_up() { + exec "${KIND_SCRIPTS}/up.sh" +} + +cluster_down() { + exec "${KIND_SCRIPTS}/down.sh" +} + +cluster_teardown() { + exec "${KIND_SCRIPTS}/teardown.sh" +} + +cluster_status() { + exec "${KIND_SCRIPTS}/status.sh" +} + +component_swap() { + exec "${KIND_SCRIPTS}/swap-component.sh" up "$1" +} + +component_revert() { + exec "${KIND_SCRIPTS}/swap-component.sh" down "$1" +} diff --git a/scripts/cluster/drivers/openshift.sh b/scripts/cluster/drivers/openshift.sh new file mode 100755 index 00000000..5e1ed152 --- /dev/null +++ b/scripts/cluster/drivers/openshift.sh @@ -0,0 +1,1287 @@ +#!/usr/bin/env bash +# OpenShift lifecycle driver. Deploys into an ephemeral namespace group on an +# existing cluster; it never creates or destroys the cluster. +set -euo pipefail + +OWNED_LABEL="hypershell.redhat.io/owned" +ENV_LABEL="hypershell.redhat.io/environment" +MANAGED_LABEL="app.kubernetes.io/managed-by" +MANAGED_VALUE="hypershell-lifecycle" +PART_OF_LABEL="app.kubernetes.io/part-of" +PART_OF_VALUE="hypershell" + +oc_cli() { + oc "$@" +} + +require_openshift_cluster() { + if ! command -v oc >/dev/null 2>&1; then + error "oc is not installed. Install the OpenShift CLI and retry." + error "Provide an OpenShift cluster target before running 'make openshift-up'." + exit 1 + fi + if ! oc_cli whoami >/dev/null 2>&1; then + error "No reachable OpenShift cluster in the current kubeconfig context." + error "Log in with 'oc login' (or set KUBECONFIG) and retry. 'make openshift-up' does not create a cluster." + exit 1 + fi + if ! oc_cli api-resources --api-group=route.openshift.io --no-headers 2>/dev/null | grep -q .; then + error "The current kubeconfig context is not an OpenShift cluster (route.openshift.io is missing)." + error "Provide an OpenShift cluster target before running 'make openshift-up'." + exit 1 + fi +} + +resolve_openshift_namespace() { + # Default: the current oc project. OPENSHIFT_NAMESPACE is an override. + if [[ -n "${OPENSHIFT_NAMESPACE:-}" ]]; then + return 0 + fi + local project + project="$(oc_cli project -q 2>/dev/null || true)" + if [[ -z "${project}" ]]; then + error "OPENSHIFT_NAMESPACE is unset and no oc project is selected." + error "Run 'oc project ' or set OPENSHIFT_NAMESPACE to a unique RFC 1123 DNS label (max 54 characters)." + exit 1 + fi + OPENSHIFT_NAMESPACE="${project}" + info "OPENSHIFT_NAMESPACE unset; using oc project '${OPENSHIFT_NAMESPACE}'" +} + +validate_namespace_group() { + validate_rfc1123_label "${OPENSHIFT_NAMESPACE}" 54 || exit 1 + OPENSHIFT_KEYCLOAK_NAMESPACE="$(keycloak_namespace_for "${OPENSHIFT_NAMESPACE}")" + validate_rfc1123_label "${OPENSHIFT_KEYCLOAK_NAMESPACE}" 63 || exit 1 + if is_reserved_cluster_namespace "${OPENSHIFT_NAMESPACE}"; then + error "Namespace '${OPENSHIFT_NAMESPACE}' is a reserved cluster namespace. Choose a different project." + exit 1 + fi + if is_reserved_cluster_namespace "${OPENSHIFT_KEYCLOAK_NAMESPACE}"; then + error "Derived Keycloak namespace '${OPENSHIFT_KEYCLOAK_NAMESPACE}' is reserved. Choose a different platform name." + exit 1 + fi +} + +namespace_exists() { + local out + if ! out="$(oc_cli get project "$1" --ignore-not-found -o name 2>/dev/null)"; then + error "checking for project $1 failed" + exit 1 + fi + [[ -n "${out}" ]] +} + +current_project() { + oc_cli project -q 2>/dev/null || true +} + +use_project() { + local ns="$1" + local current + current="$(current_project)" + if [[ "${current}" == "${ns}" ]]; then + return 0 + fi + info "Using oc project ${ns}" + if oc_cli project "${ns}" >/dev/null; then + return 0 + fi + error "Cannot switch to project ${ns}" + exit 1 +} + +namespace_label_value() { + oc_cli get namespace "$1" -o go-template="{{if .metadata.labels}}{{index .metadata.labels \"$2\"}}{{end}}" 2>/dev/null || true +} + +namespace_is_owned() { + [[ "$(namespace_label_value "$1" "${OWNED_LABEL}")" == "true" ]] +} + +stamp_namespace_labels() { + local ns="$1" + local env_id="$2" + local err + if err="$(oc_cli label namespace "${ns}" \ + "${OWNED_LABEL}=true" \ + "${ENV_LABEL}=${env_id}" \ + "${MANAGED_LABEL}=${MANAGED_VALUE}" \ + "${PART_OF_LABEL}=${PART_OF_VALUE}" \ + --overwrite 2>&1)"; then + return 0 + fi + if grep -qi 'forbidden' <<<"${err}"; then + warn "Cannot label namespace ${ns} (no permission to patch namespaces). Continuing." + return 0 + fi + error "Failed to label namespace ${ns}: ${err}" + exit 1 +} + +# Namespaces labeled as a different HyperShell environment are foreign. +# Unlabeled namespaces are the developer's chosen target, not foreign. +refuse_foreign_namespace() { + local ns="$1" + local expected_env="${2:-}" + if ! namespace_exists "${ns}"; then + return 0 + fi + local owned env_id + owned="$(namespace_label_value "${ns}" "${OWNED_LABEL}")" + env_id="$(namespace_label_value "${ns}" "${ENV_LABEL}")" + if [[ "${owned}" != "true" ]]; then + return 0 + fi + if [[ -z "${env_id}" ]]; then + error "Namespace '${ns}' is missing the ${ENV_LABEL} identifier. Refusing to adopt it." + exit 1 + fi + if [[ -n "${expected_env}" && "${env_id}" != "${expected_env}" ]]; then + error "Namespace '${ns}' belongs to environment '${env_id}', not '${expected_env}'. Refusing to adopt it." + exit 1 + fi + printf '%s' "${env_id}" +} + +# Recover a previously applied environment id from workload labels when the +# namespace itself could not be labeled. +env_id_from_workloads() { + local ns="$1" + oc_cli get deploy -n "${ns}" -o go-template \ + "{{range .items}}{{if .metadata.labels}}{{with index .metadata.labels \"${ENV_LABEL}\"}}{{.}}{{println}}{{end}}{{end}}{{end}}" \ + 2>/dev/null | awk 'NF{print; exit}' || true +} + +ensure_project() { + local ns="$1" + local env_id="$2" + if namespace_exists "${ns}"; then + if namespace_is_owned "${ns}"; then + refuse_foreign_namespace "${ns}" "${env_id}" >/dev/null + fi + stamp_namespace_labels "${ns}" "${env_id}" + return 0 + fi + info "Creating project ${ns} (oc new-project)" + local err + if err="$(oc_cli new-project "${ns}" \ + --display-name="HyperShell ${ns}" \ + --description="HyperShell local-dev" 2>&1)"; then + stamp_namespace_labels "${ns}" "${env_id}" + return 0 + fi + if grep -qi 'already exists' <<<"${err}"; then + use_project "${ns}" + stamp_namespace_labels "${ns}" "${env_id}" + return 0 + fi + if grep -qi 'forbidden' <<<"${err}"; then + error "Cannot create project ${ns} (forbidden)." + error "OpenShift developers create projects with 'oc new-project', not 'oc create namespace'." + error "Ask an administrator to grant self-provisioner, or to create ${ns}." + exit 1 + fi + error "Failed to create project ${ns}: ${err}" + exit 1 +} + +ensure_namespace_group() { + local platform_env keycloak_env env_id + platform_env="" + keycloak_env="" + if namespace_exists "${OPENSHIFT_NAMESPACE}" && namespace_is_owned "${OPENSHIFT_NAMESPACE}"; then + platform_env="$(refuse_foreign_namespace "${OPENSHIFT_NAMESPACE}")" + fi + if namespace_exists "${OPENSHIFT_KEYCLOAK_NAMESPACE}" && namespace_is_owned "${OPENSHIFT_KEYCLOAK_NAMESPACE}"; then + keycloak_env="$(refuse_foreign_namespace "${OPENSHIFT_KEYCLOAK_NAMESPACE}" "${platform_env}")" + fi + if [[ -n "${platform_env}" && -n "${keycloak_env}" && "${platform_env}" != "${keycloak_env}" ]]; then + error "Namespace group mismatch: ${OPENSHIFT_NAMESPACE}=${platform_env} vs ${OPENSHIFT_KEYCLOAK_NAMESPACE}=${keycloak_env}" + exit 1 + fi + env_id="${platform_env:-${keycloak_env}}" + if [[ -z "${env_id}" ]]; then + env_id="$(env_id_from_workloads "${OPENSHIFT_NAMESPACE}")" + fi + if [[ -z "${env_id}" ]]; then + env_id="$(uuidgen | tr '[:upper:]' '[:lower:]')" + fi + OPENSHIFT_ENVIRONMENT_ID="${env_id}" + ensure_project "${OPENSHIFT_NAMESPACE}" "${env_id}" + ensure_project "${OPENSHIFT_KEYCLOAK_NAMESPACE}" "${env_id}" + use_project "${OPENSHIFT_NAMESPACE}" + success "Namespace group ${OPENSHIFT_NAMESPACE} + ${OPENSHIFT_KEYCLOAK_NAMESPACE} (environment ${env_id})" +} + +discover_gateway_base_domain() { + local gw_name="$1" + local gw_ns="$2" + local host="" + host="$(oc_cli get "gateway.gateway.networking.k8s.io/${gw_name}" -n "${gw_ns}" \ + -o jsonpath='{.spec.listeners[?(@.name=="grpc")].hostname}' 2>/dev/null || true)" + if [[ -z "${host}" ]]; then + host="$(oc_cli get "gateway.gateway.networking.k8s.io/${gw_name}" -n "${gw_ns}" \ + -o jsonpath='{.spec.listeners[0].hostname}' 2>/dev/null || true)" + fi + if [[ -z "${host}" ]]; then + error "Shared Gateway '${gw_ns}/${gw_name}' has no listener hostname." + error "The gateway wildcard hostname is the base domain; 'make openshift-up' does not take GATEWAY_API_BASE_DOMAIN." + exit 1 + fi + GATEWAY_API_BASE_DOMAIN="$(gateway_base_domain_from_hostname "${host}")" + if [[ -z "${GATEWAY_API_BASE_DOMAIN}" ]]; then + error "Could not derive a base domain from Gateway listener hostname '${host}'." + exit 1 + fi +} + +check_infrastructure() { + header "Infrastructure prerequisites" + local gw_name="${GATEWAY_API_GATEWAY_NAME}" + local gw_ns="${GATEWAY_API_GATEWAY_NAMESPACE}" + if ! oc_cli get "gateway.gateway.networking.k8s.io/${gw_name}" -n "${gw_ns}" >/dev/null 2>&1; then + error "Required shared Gateway '${gw_name}' was not found in namespace '${gw_ns}'." + error "An administrator must provision the Gateway, GatewayClass, certificate issuer, and wildcard certificate first." + error "See deploy/openshift/infrastructure/GATEWAY-SETUP.md. 'make openshift-up' will not create cluster infrastructure." + exit 1 + fi + local gw_class + gw_class="$(oc_cli get "gateway.gateway.networking.k8s.io/${gw_name}" -n "${gw_ns}" -o jsonpath='{.spec.gatewayClassName}' 2>/dev/null || true)" + if [[ -z "${gw_class}" ]]; then + error "Shared Gateway '${gw_ns}/${gw_name}' has no gatewayClassName." + exit 1 + fi + # GatewayClass is cluster-scoped. Typical developers cannot GET it; a + # Programmed Gateway is the proof the class is serving this cluster. + local programmed + programmed="$(oc_cli get "gateway.gateway.networking.k8s.io/${gw_name}" -n "${gw_ns}" \ + -o jsonpath='{range .status.conditions[?(@.type=="Programmed")]}{.status}{end}' 2>/dev/null || true)" + if [[ "${programmed}" != "True" ]]; then + error "Shared Gateway '${gw_ns}/${gw_name}' is not Programmed=True (status=${programmed:-unknown})." + error "Fix the cluster infrastructure before deploying HyperShell." + exit 1 + fi + discover_gateway_base_domain "${gw_name}" "${gw_ns}" + success "Shared Gateway ${gw_ns}/${gw_name} (GatewayClass ${gw_class}) is ready" + info "Gateway base domain: ${GATEWAY_API_BASE_DOMAIN} (from ${gw_ns}/${gw_name} listener)" +} + +apply_bind_grant() { + # Privileged local-dev actor: grant this environment's controller SA the + # cluster-scoped bind on the privileged SCC. Typical developers cannot + # create ClusterRoleBindings; skip rather than fail the whole bring-up. + local err + if err="$(oc_cli apply -f - 2>&1 </dev/null + + oc_cli create secret generic hypershell-api-config \ + -n "${OPENSHIFT_NAMESPACE}" \ + --from-literal=api-service.issuerUrl="${kc_svc}/realms/hypershell" \ + --from-literal=api-service.clientId="hypershell-control-plane" \ + --from-literal=api-service.clientSecret="control-plane-secret" \ + --from-literal=api-service.jwkCertUrl="${kc_svc}/realms/hypershell/protocol/openid-connect/certs" \ + --dry-run=client -o yaml | oc_cli apply -f - >/dev/null + + oc_cli create secret generic hypershell-cp-oidc \ + -n "${OPENSHIFT_NAMESPACE}" \ + --from-literal=client-secret="control-plane-secret" \ + --dry-run=client -o yaml | oc_cli apply -f - >/dev/null + + oc_cli create secret generic hypershell-keycloak-admin \ + -n "${OPENSHIFT_NAMESPACE}" \ + --from-literal=server-url="${kc_svc}" \ + --from-literal=realm="hypershell" \ + --from-literal=client-id="hypershell-provisioner" \ + --from-literal=client-secret="provisioner-secret" \ + --dry-run=client -o yaml | oc_cli apply -f - >/dev/null +} + +render_openshift_manifests() { + local tmp rendered overlay_rel + tmp="$(mktemp -d "${REPO_ROOT}/.openshift-render.XXXXXX")" + overlay_rel="$(python3 -c 'import os, sys; print(os.path.relpath(sys.argv[1], sys.argv[2]))' \ + "${REPO_ROOT}/deploy/openshift" "${tmp}")" + cat > "${tmp}/kustomization.yaml" < "${rendered}"; then + rm -rf "${tmp}" + rm -f "${rendered}" + error "Failed to render OpenShift overlay" + return 1 + fi + rm -rf "${tmp}" + if [[ ! -s "${rendered}" ]]; then + rm -f "${rendered}" + error "Rendered overlay is empty" + return 1 + fi + printf '%s' "${rendered}" +} + +api_group_available() { + oc_cli api-resources --api-group="$1" --no-headers 2>/dev/null | grep -q . +} + +cnpg_available() { + api_group_available postgresql.cnpg.io +} + +apply_rendered_overlay() { + local rendered="$1" + local prune_args=( + --prune + -l "${ENV_LABEL}=${OPENSHIFT_ENVIRONMENT_ID}" + --prune-allowlist=core/v1/ConfigMap + --prune-allowlist=core/v1/Service + --prune-allowlist=core/v1/ServiceAccount + --prune-allowlist=apps/v1/Deployment + --prune-allowlist=networking.k8s.io/v1/NetworkPolicy + --prune-allowlist=route.openshift.io/v1/Route + --prune-allowlist=rbac.authorization.k8s.io/v1/RoleBinding + ) + if api_group_available cert-manager.io; then + prune_args+=( + --prune-allowlist=cert-manager.io/v1/Certificate + --prune-allowlist=cert-manager.io/v1/Issuer + ) + fi + if cnpg_available; then + prune_args+=(--prune-allowlist=postgresql.cnpg.io/v1/Cluster) + fi + # Secrets are omitted from prune: bootstrap OIDC secrets are not in the overlay + # and must survive reconcile. A reconcile that drops a swapped Deployment's + # image is restored by restore_swaps_after_reconcile. + if [[ ! -s "${rendered}" ]]; then + error "Rendered overlay is empty" + rm -f "${rendered}" + exit 1 + fi + if ! oc_cli apply -f "${rendered}" "${prune_args[@]}"; then + warn "Apply with prune failed; applying without prune" + oc_cli apply -f "${rendered}" + fi + rm -f "${rendered}" +} + +apply_optional() { + local rendered="$1" + local what="$2" + if [[ ! -s "${rendered}" ]]; then + rm -f "${rendered}" + return 0 + fi + local err + if err="$(oc_cli apply -f "${rendered}" 2>&1)"; then + printf '%s\n' "${err}" + rm -f "${rendered}" + return 0 + fi + rm -f "${rendered}" + if grep -qi 'forbidden' <<<"${err}"; then + warn "Skipping ${what}: the current user cannot apply cluster-scoped or privileged RBAC." + warn "An administrator must grant this namespace's hypershell-controller SA the shared ClusterRole and privileged SCC bind." + return 0 + fi + error "${err}" + return 1 +} + +apply_postgres_fallback() { + info "CNPG operator is not installed; deploying bundled PostgreSQL Deployment" + local rendered + rendered="$(mktemp)" + if ! python3 "${CLUSTER_SCRIPT_DIR}/rewrite-namespaces.py" \ + --platform-namespace "${OPENSHIFT_NAMESPACE}" \ + --keycloak-namespace "${OPENSHIFT_KEYCLOAK_NAMESPACE}" \ + --omit-namespaces \ + --only-namespace "${OPENSHIFT_NAMESPACE}" \ + --strip-openshift-uids \ + < "${REPO_ROOT}/deploy/base/postgres.yaml" \ + > "${rendered}"; then + rm -f "${rendered}" + error "Failed to render PostgreSQL fallback" + return 1 + fi + oc_cli apply -f "${rendered}" + rm -f "${rendered}" +} + +configure_postgres_fallback_ssl() { + oc_cli set env deployment/hypershell-api-server -n "${OPENSHIFT_NAMESPACE}" -c migrate \ + DB_SSLMODE=disable >/dev/null + oc_cli set env deployment/hypershell-api-server -n "${OPENSHIFT_NAMESPACE}" -c api-server \ + DB_SSLMODE=disable >/dev/null +} + +developer_omit_kinds() { + local kinds="ClusterRole,ClusterRoleBinding" + if ! cnpg_available; then + kinds+=",Cluster" + fi + if ! api_group_available cert-manager.io; then + kinds+=",Certificate,Issuer" + fi + printf '%s' "${kinds}" +} + +apply_overlay() { + header "Deploying Components" + local rendered omit_kinds + omit_kinds="$(developer_omit_kinds)" + + info "Applying Keycloak in project ${OPENSHIFT_KEYCLOAK_NAMESPACE}..." + use_project "${OPENSHIFT_KEYCLOAK_NAMESPACE}" + if ! rendered="$(render_openshift_manifests --only-namespace "${OPENSHIFT_KEYCLOAK_NAMESPACE}")"; then + exit 1 + fi + apply_rendered_overlay "${rendered}" + + info "Applying HyperShell in project ${OPENSHIFT_NAMESPACE}..." + use_project "${OPENSHIFT_NAMESPACE}" + if ! cnpg_available; then + apply_postgres_fallback + fi + if ! rendered="$(render_openshift_manifests \ + --only-namespace "${OPENSHIFT_NAMESPACE}" \ + --omit-kinds "${omit_kinds}" \ + --omit-names hypershell-sandbox-scc)"; then + exit 1 + fi + apply_rendered_overlay "${rendered}" + if ! cnpg_available; then + configure_postgres_fallback_ssl + fi + + info "Applying optional cluster-scoped RBAC (best-effort)..." + if rendered="$(render_openshift_manifests --only-namespace __cluster__ --include-cluster-scoped)"; then + apply_optional "${rendered}" "ClusterRole / ClusterRoleBinding" + fi + apply_sandbox_scc_best_effort + + success "Overlay applied" +} + +apply_sandbox_scc_best_effort() { + local err + if err="$(oc_cli apply -n "${OPENSHIFT_NAMESPACE}" -f - 2>&1 </dev/null || true)" + if [[ -n "${host}" ]]; then + printf '%s' "${host}" + return 0 + fi + sleep 2 + done + return 1 +} + +wait_for_keycloak() { + info "Waiting for Keycloak..." + oc_cli rollout status deployment/keycloak -n "${OPENSHIFT_KEYCLOAK_NAMESPACE}" --timeout=180s + success "Keycloak ready" +} + +configure_oidc_from_routes() { + header "OIDC" + wait_for_keycloak + info "Waiting for Keycloak Route..." + local kc_host api_host console_host + if ! kc_host="$(wait_for_route_host keycloak "${OPENSHIFT_KEYCLOAK_NAMESPACE}")"; then + error "Keycloak Route host was not assigned in ${OPENSHIFT_KEYCLOAK_NAMESPACE}" + exit 1 + fi + if ! api_host="$(wait_for_route_host hypershell-api "${OPENSHIFT_NAMESPACE}")"; then + error "API Route host was not assigned in ${OPENSHIFT_NAMESPACE}" + exit 1 + fi + if ! console_host="$(wait_for_route_host hypershell-web-console "${OPENSHIFT_NAMESPACE}")"; then + error "Web console Route host was not assigned in ${OPENSHIFT_NAMESPACE}" + exit 1 + fi + OPENSHIFT_KEYCLOAK_HOST="${kc_host}" + OPENSHIFT_API_HOST="${api_host}" + OPENSHIFT_CONSOLE_HOST="${console_host}" + OPENSHIFT_KC_HOSTNAME="https://${kc_host}" + OPENSHIFT_OIDC_ISSUER="https://${kc_host}/realms/hypershell" + + info "Setting Keycloak KC_HOSTNAME=${OPENSHIFT_KC_HOSTNAME}" + oc_cli set env deployment/keycloak -n "${OPENSHIFT_KEYCLOAK_NAMESPACE}" \ + "KC_HOSTNAME=${OPENSHIFT_KC_HOSTNAME}" >/dev/null + + info "Configuring API server JWT environment" + oc_cli set env deployment/hypershell-api-server -n "${OPENSHIFT_NAMESPACE}" -c api-server \ + API_ENV=development_oidc >/dev/null + + info "Configuring web console OIDC" + oc_cli set env deployment/hypershell-web-console -n "${OPENSHIFT_NAMESPACE}" -c web-console \ + "OIDC_ISSUER=${OPENSHIFT_OIDC_ISSUER}" \ + OIDC_CLIENT_ID=hypershell-frontend \ + "OIDC_REDIRECT_URI=https://${console_host}/auth/callback" \ + "OIDC_POST_LOGOUT_REDIRECT_URI=https://${console_host}" >/dev/null + oc_cli set env deployment/hypershell-web-console -n "${OPENSHIFT_NAMESPACE}" -c web-console \ + --from=secret/hypershell-oidc-session >/dev/null + + info "Configuring control plane public gateway issuer" + oc_cli set env deployment/hypershell-controller -n "${OPENSHIFT_NAMESPACE}" -c controller \ + "GATEWAY_OIDC_ISSUER_URL=${OPENSHIFT_OIDC_ISSUER}" >/dev/null + + success "Routes: api=${api_host} console=${console_host} keycloak=${kc_host}" +} + +wait_for_deployments() { + header "Readiness" + wait_for_keycloak + + if [[ -n "$(oc_cli get cluster.postgresql.cnpg.io hypershell-db -n "${OPENSHIFT_NAMESPACE}" --ignore-not-found -o name 2>/dev/null || true)" ]]; then + info "Waiting for CNPG cluster..." + oc_cli wait --for=condition=Ready cluster/hypershell-db -n "${OPENSHIFT_NAMESPACE}" --timeout=300s \ + || warn "CNPG cluster not Ready yet; API server will retry connections" + elif oc_cli get deployment/hypershell-postgres -n "${OPENSHIFT_NAMESPACE}" >/dev/null 2>&1; then + info "Waiting for PostgreSQL deployment..." + if ! oc_cli rollout status deployment/hypershell-postgres -n "${OPENSHIFT_NAMESPACE}" --timeout=120s; then + warn "PostgreSQL rollout timed out. ReplicaSet events:" + oc_cli describe deploy/hypershell-postgres -n "${OPENSHIFT_NAMESPACE}" | tail -20 || true + oc_cli get events -n "${OPENSHIFT_NAMESPACE}" --field-selector involvedObject.kind=ReplicaSet \ + --sort-by='.lastTimestamp' 2>/dev/null | tail -10 || true + exit 1 + fi + success "PostgreSQL ready" + fi + + if ! is_openshift_swapped api-server; then + info "Waiting for API server..." + oc_cli rollout status deployment/hypershell-api-server -n "${OPENSHIFT_NAMESPACE}" --timeout=180s + success "API server ready" + fi + if ! is_openshift_swapped control-plane; then + info "Waiting for control plane..." + oc_cli rollout status deployment/hypershell-controller -n "${OPENSHIFT_NAMESPACE}" --timeout=180s + success "Control plane ready" + fi + if ! is_openshift_swapped web-console; then + info "Waiting for web console..." + oc_cli rollout status deployment/hypershell-web-console -n "${OPENSHIFT_NAMESPACE}" --timeout=180s + success "Web console ready" + fi +} + +# Talk to OpenShift Routes from the developer machine. The API server image has +# no curl, so oc exec cannot reach Keycloak or the API. -k matches kind-up: +# cluster default certs are not always in the local trust store. +openshift_curl() { + curl -sS -k -m 15 "$@" +} + +add_keycloak_redirect_uri() { + if ! command -v curl >/dev/null 2>&1; then + warn "curl is required on this machine to register the console redirect URI in Keycloak" + return 0 + fi + local callback="https://${OPENSHIFT_CONSOLE_HOST}/auth/callback" + local kc="${OPENSHIFT_KC_HOSTNAME}" + local token client_json client_id full updated http + token="$(openshift_curl -X POST "${kc}/realms/master/protocol/openid-connect/token" \ + -d grant_type=password -d client_id=admin-cli \ + -d username=admin -d password=admin \ + | json_string_field access_token || true)" + if [[ -z "${token}" ]]; then + warn "Could not obtain Keycloak admin token to add console redirect URI" + return 0 + fi + client_json="$(openshift_curl -H "Authorization: Bearer ${token}" \ + "${kc}/admin/realms/hypershell/clients?clientId=hypershell-frontend" || true)" + client_id="$(printf '%s' "${client_json}" | json_first_id || true)" + if [[ -z "${client_id}" ]]; then + warn "Could not find hypershell-frontend client to add redirect URI" + return 0 + fi + full="$(openshift_curl -H "Authorization: Bearer ${token}" \ + "${kc}/admin/realms/hypershell/clients/${client_id}" || true)" + if [[ -z "${full}" ]]; then + warn "Could not fetch hypershell-frontend client representation" + return 0 + fi + if printf '%s' "${full}" | grep -Fq "${callback}"; then + info "Keycloak redirect URI already includes ${callback}" + return 0 + fi + info "Setting Keycloak redirect URIs for ${OPENSHIFT_CONSOLE_HOST}" + updated="$(printf '%s' "${full}" | keycloak_client_with_console_redirects "${OPENSHIFT_CONSOLE_HOST}")" + http="$(openshift_curl -o /dev/null -w '%{http_code}' -X PUT \ + -H "Authorization: Bearer ${token}" \ + -H "Content-Type: application/json" \ + "${kc}/admin/realms/hypershell/clients/${client_id}" \ + -d "${updated}" || true)" + if [[ "${http}" != "204" && "${http}" != "200" ]]; then + warn "Failed to update Keycloak redirect URIs (HTTP ${http:-none})" + return 0 + fi + success "Keycloak redirect URI ${callback}" +} + +seed_via_api() { + header "Gateway Provisioning" + local kc_token_url="${OPENSHIFT_OIDC_ISSUER}/protocol/openid-connect/token" + local api="https://${OPENSHIFT_API_HOST}" + local token="" resp="" + local i + if ! command -v curl >/dev/null 2>&1; then + warn "curl is required on this machine to seed the API; skip automatic seeding" + return 0 + fi + info "Obtaining API token from Keycloak Route..." + for i in $(seq 1 30); do + resp="$(openshift_curl -X POST "${kc_token_url}" \ + -d grant_type=password -d client_id=hypershell-frontend \ + -d username=admin -d password=admin || true)" + token="$(printf '%s' "${resp}" | json_string_field access_token || true)" + if [[ -n "${token}" ]]; then + break + fi + sleep 2 + done + if [[ -z "${token}" ]]; then + warn "Could not obtain API token; skip automatic seeding" + return 0 + fi + success "API token obtained" + + api_exec() { + local method="$1" path="$2" data="${3:-}" + if [[ -n "${data}" ]]; then + openshift_curl -w "\n%{http_code}" -X "${method}" "${api}${path}" \ + -H "Authorization: Bearer ${token}" \ + -H "Content-Type: application/json" \ + -d "${data}" || true + else + openshift_curl -w "\n%{http_code}" -X "${method}" "${api}${path}" \ + -H "Authorization: Bearer ${token}" || true + fi + } + + extract_named_id() { + local resp="$1" name="$2" + printf '%s' "${resp}" | grep -o "\"name\":\"${name}\"[^}]*\"id\":\"[^\"]*\"" \ + | grep -o '"id":"[^"]*"' | cut -d'"' -f4 | head -1 || true + } + + extract_id() { + local resp="$1" + if echo "${resp}" | grep -q '"kind":"Error"'; then + echo "" + return + fi + echo "${resp}" | grep -o '"id":"[^"]*"' | head -1 | cut -d'"' -f4 || true + } + + local seed_failed="" FLEET_ID="" CLUSTER_ID="" RELEASE_ID="" DATABASE_ID="" GATEWAY_ID="" + local raw http body + + raw="$(api_exec GET /api/hypershell/v1/fleets)" + http="$(printf '%s' "${raw}" | tail -1)" + body="$(printf '%s' "${raw}" | sed '$d')" + if [[ "${http}" == "200" ]]; then + FLEET_ID="$(extract_named_id "${body}" default)" + fi + if [[ -z "${FLEET_ID}" ]]; then + info "Creating default Fleet..." + raw="$(api_exec POST /api/hypershell/v1/fleets '{"name":"default","description":"OpenShift development fleet"}')" + http="$(printf '%s' "${raw}" | tail -1)" + body="$(printf '%s' "${raw}" | sed '$d')" + FLEET_ID="$(extract_id "${body}")" + if [[ -z "${FLEET_ID}" ]]; then + warn "Fleet creation failed (HTTP ${http}): ${body:-no response}" + seed_failed=true + else + success "Fleet created: ${FLEET_ID}" + fi + else + success "default Fleet already exists: ${FLEET_ID}" + fi + + if [[ -z "${seed_failed}" ]]; then + raw="$(api_exec GET /api/hypershell/v1/managed_clusters)" + http="$(printf '%s' "${raw}" | tail -1)" + body="$(printf '%s' "${raw}" | sed '$d')" + if [[ "${http}" == "200" ]]; then + CLUSTER_ID="$(extract_named_id "${body}" local-openshift)" + fi + if [[ -z "${CLUSTER_ID}" ]]; then + info "Creating ManagedCluster..." + raw="$(api_exec POST /api/hypershell/v1/managed_clusters \ + "{\"name\":\"local-openshift\",\"fleet_id\":\"${FLEET_ID}\",\"provider\":\"openshift\",\"kubeconfig_secret\":\"openshift-kubeconfig\"}")" + http="$(printf '%s' "${raw}" | tail -1)" + body="$(printf '%s' "${raw}" | sed '$d')" + CLUSTER_ID="$(extract_id "${body}")" + if [[ -z "${CLUSTER_ID}" ]]; then + warn "ManagedCluster creation failed (HTTP ${http}): ${body:-no response}" + seed_failed=true + else + success "ManagedCluster created: ${CLUSTER_ID}" + fi + else + success "local-openshift ManagedCluster already exists: ${CLUSTER_ID}" + fi + fi + + if [[ -z "${seed_failed}" ]]; then + raw="$(api_exec GET /api/hypershell/v1/gateway_releases)" + http="$(printf '%s' "${raw}" | tail -1)" + body="$(printf '%s' "${raw}" | sed '$d')" + if [[ "${http}" == "200" ]]; then + RELEASE_ID="$(extract_named_id "${body}" dev-release)" + fi + if [[ -z "${RELEASE_ID}" ]]; then + info "Creating GatewayRelease..." + raw="$(api_exec POST /api/hypershell/v1/gateway_releases \ + "{\"name\":\"dev-release\",\"fleet_id\":\"${FLEET_ID}\",\"image\":\"${GATEWAY_IMAGE}\"}")" + http="$(printf '%s' "${raw}" | tail -1)" + body="$(printf '%s' "${raw}" | sed '$d')" + RELEASE_ID="$(extract_id "${body}")" + if [[ -z "${RELEASE_ID}" ]]; then + warn "GatewayRelease creation failed (HTTP ${http}): ${body:-no response}" + seed_failed=true + else + success "GatewayRelease created: ${RELEASE_ID}" + fi + else + success "dev-release GatewayRelease already exists: ${RELEASE_ID}" + fi + fi + + if [[ -z "${seed_failed}" ]]; then + raw="$(api_exec GET /api/hypershell/v1/managed_databases)" + http="$(printf '%s' "${raw}" | tail -1)" + body="$(printf '%s' "${raw}" | sed '$d')" + if [[ "${http}" == "200" ]]; then + DATABASE_ID="$(extract_named_id "${body}" openshell-db)" + fi + if [[ -z "${DATABASE_ID}" ]]; then + info "Creating ManagedDatabase..." + raw="$(api_exec POST /api/hypershell/v1/managed_databases \ + "{\"name\":\"openshell-db\",\"fleet_id\":\"${FLEET_ID}\",\"provider\":\"cnpg\"}")" + http="$(printf '%s' "${raw}" | tail -1)" + body="$(printf '%s' "${raw}" | sed '$d')" + if [[ "${http}" != "201" && "${http}" != "200" ]]; then + warn "ManagedDatabase creation failed (HTTP ${http}): ${body:-no response}" + seed_failed=true + else + DATABASE_ID="$(extract_id "${body}")" + success "ManagedDatabase created: ${DATABASE_ID}" + fi + else + success "openshell-db ManagedDatabase already exists: ${DATABASE_ID}" + fi + fi + + if [[ -z "${seed_failed}" ]]; then + raw="$(api_exec GET /api/hypershell/v1/gateways)" + http="$(printf '%s' "${raw}" | tail -1)" + body="$(printf '%s' "${raw}" | sed '$d')" + if [[ "${http}" == "200" ]]; then + GATEWAY_ID="$(extract_named_id "${body}" dev-gateway)" + fi + if [[ -z "${GATEWAY_ID}" ]]; then + info "Creating Gateway with OIDC..." + local oidc + oidc="{\\\"issuer\\\":\\\"${OPENSHIFT_OIDC_ISSUER}\\\",\\\"audience\\\":\\\"hypershell-frontend\\\",\\\"roles_claim\\\":\\\"groups\\\",\\\"admin_role\\\":\\\"hypershell-admins\\\",\\\"user_role\\\":\\\"hypershell-users\\\"}" + raw="$(api_exec POST /api/hypershell/v1/gateways \ + "{\"name\":\"dev-gateway\",\"fleet_id\":\"${FLEET_ID}\",\"cluster_id\":\"${CLUSTER_ID}\",\"release_id\":\"${RELEASE_ID}\",\"database_id\":\"${DATABASE_ID}\",\"oidc\":\"${oidc}\"}")" + http="$(printf '%s' "${raw}" | tail -1)" + body="$(printf '%s' "${raw}" | sed '$d')" + GATEWAY_ID="$(extract_id "${body}")" + if [[ -z "${GATEWAY_ID}" ]]; then + warn "Gateway creation failed (HTTP ${http}): ${body:-no response}" + else + success "Gateway created: ${GATEWAY_ID}" + fi + else + success "dev-gateway already exists: ${GATEWAY_ID}" + fi + fi + + if [[ -n "${seed_failed}" ]]; then + warn "Automatic seeding incomplete - create resources manually after the API server is ready" + fi +} + +print_banner() { + header "HyperShell is running on OpenShift" + echo "" + info "Namespace: ${OPENSHIFT_NAMESPACE} (Keycloak: ${OPENSHIFT_KEYCLOAK_NAMESPACE})" + info "HTTP API: https://${OPENSHIFT_API_HOST}" + info "Web Console: https://${OPENSHIFT_CONSOLE_HOST}" + info "Keycloak: ${OPENSHIFT_KC_HOSTNAME} (admin/admin)" + info "OIDC Issuer: ${OPENSHIFT_OIDC_ISSUER}" + info "Login: https://${OPENSHIFT_CONSOLE_HOST}/auth/login" + info "Test users: admin/admin (admins + users), developer/developer (users only)" + echo "" + info "API Server Logs: oc logs -f -l app=hypershell-api-server -n ${OPENSHIFT_NAMESPACE}" + info "Control Plane Logs: oc logs -f -l app=hypershell-controller -n ${OPENSHIFT_NAMESPACE}" + info "Web Console Logs: oc logs -f -l app=hypershell-web-console -n ${OPENSHIFT_NAMESPACE}" +} + +cluster_up() { + header "HyperShell OpenShift Development Environment" + echo "" + require_openshift_cluster + resolve_openshift_namespace + OPENSHIFT_ENTRY_PROJECT="$(current_project)" + trap 'if [[ -n "${OPENSHIFT_ENTRY_PROJECT:-}" ]]; then oc_cli project "${OPENSHIFT_ENTRY_PROJECT}" >/dev/null 2>&1 || true; fi' EXIT + validate_namespace_group + check_infrastructure + ensure_namespace_group + create_bootstrap_secrets + apply_overlay + apply_bind_grant + restore_swaps_after_reconcile + configure_oidc_from_routes + wait_for_deployments + add_keycloak_redirect_uri || true + seed_via_api + echo "" + print_banner +} + +verify_owned_namespace() { + local ns="$1" + if ! namespace_exists "${ns}"; then + warn "Namespace ${ns} does not exist" + return 1 + fi + local owned env_id + owned="$(namespace_label_value "${ns}" "${OWNED_LABEL}")" + env_id="$(namespace_label_value "${ns}" "${ENV_LABEL}")" + if [[ "${owned}" == "true" && -n "${env_id}" && -n "${OPENSHIFT_ENVIRONMENT_ID:-}" && "${env_id}" != "${OPENSHIFT_ENVIRONMENT_ID}" ]]; then + error "Namespace '${ns}' belongs to environment '${env_id}', not '${OPENSHIFT_ENVIRONMENT_ID}'. Refusing to delete it." + exit 1 + fi + if [[ "${owned}" == "true" && -n "${env_id}" ]]; then + OPENSHIFT_ENVIRONMENT_ID="${env_id}" + fi +} + +delete_hypershell_resources() { + local ns="$1" + info "Removing HyperShell resources from ${ns} (keeping the project)" + oc_cli delete deploy,statefulset,job,cronjob,pod,svc,cm,secret,pvc,sa,role,rolebinding,route,networkpolicy \ + -n "${ns}" \ + -l "${MANAGED_LABEL}=${MANAGED_VALUE}" \ + --ignore-not-found --wait=true --timeout=180s >/dev/null 2>&1 || true + oc_cli delete deploy,svc,cm,secret,pvc,sa \ + -n "${ns}" \ + -l app.kubernetes.io/name=hypershell \ + --ignore-not-found --wait=true --timeout=180s >/dev/null 2>&1 || true + oc_cli delete issuer,certificate \ + -n "${ns}" \ + -l "${MANAGED_LABEL}=${MANAGED_VALUE}" \ + --ignore-not-found --wait=true --timeout=180s >/dev/null 2>&1 || true + oc_cli delete cluster.postgresql.cnpg.io \ + -n "${ns}" \ + --ignore-not-found --wait=true --timeout=180s >/dev/null 2>&1 || true + oc_cli delete deploy,svc,secret \ + -n "${ns}" \ + hypershell-postgres hypershell-db-app \ + --ignore-not-found --wait=true --timeout=180s >/dev/null 2>&1 || true + oc_cli delete secret \ + -n "${ns}" \ + hypershell-oidc-session hypershell-api-config hypershell-cp-oidc hypershell-keycloak-admin \ + --ignore-not-found --wait=true --timeout=180s >/dev/null 2>&1 || true + # Bundled Keycloak is unlabeled (deploy/base/keycloak). Delete it by name so a + # forbidden project-delete still clears the companion -keycloak namespace. + oc_cli delete deploy,svc,cm,secret,route,networkpolicy \ + -n "${ns}" \ + keycloak keycloak-service keycloak-realm \ + keycloak-hypershell-theme keycloak-hypershell-theme-assets \ + keycloak-allow-platform \ + --ignore-not-found --wait=true --timeout=180s >/dev/null 2>&1 || true +} + +wait_until_project_gone() { + local ns="$1" + local timeout="${2:-300}" + local elapsed=0 + while namespace_exists "${ns}"; do + if (( elapsed >= timeout )); then + error "Timed out after ${timeout}s waiting for project ${ns} to be deleted" + return 1 + fi + sleep 2 + elapsed=$((elapsed + 2)) + done +} + +remove_project() { + local ns="$1" + local err + if ! namespace_exists "${ns}"; then + info "Project ${ns} does not exist" + return 0 + fi + info "Deleting project ${ns}" + if err="$(oc_cli delete project "${ns}" --wait=true --timeout=300s 2>&1)"; then + wait_until_project_gone "${ns}" 60 || return 1 + success "Project ${ns} deleted" + return 0 + fi + if grep -qi 'not found' <<<"${err}"; then + return 0 + fi + if grep -qi 'forbidden' <<<"${err}"; then + warn "Cannot delete project ${ns} (forbidden)." + delete_hypershell_resources "${ns}" + return 0 + fi + error "Failed to delete project ${ns}: ${err}" + return 1 +} + +cluster_down() { + header "Removing OpenShift environment" + require_openshift_cluster + resolve_openshift_namespace + validate_namespace_group + + local ok_platform=false ok_keycloak=false + if namespace_exists "${OPENSHIFT_NAMESPACE}"; then + verify_owned_namespace "${OPENSHIFT_NAMESPACE}" + ok_platform=true + fi + if namespace_exists "${OPENSHIFT_KEYCLOAK_NAMESPACE}"; then + verify_owned_namespace "${OPENSHIFT_KEYCLOAK_NAMESPACE}" + ok_keycloak=true + fi + if [[ "${ok_platform}" != "true" && "${ok_keycloak}" != "true" ]]; then + warn "No namespace group found for ${OPENSHIFT_NAMESPACE} / ${OPENSHIFT_KEYCLOAK_NAMESPACE}" + clear_all_openshift_swaps + return 0 + fi + + info "Deleting per-namespace SCC bind ClusterRoleBinding..." + oc_cli delete clusterrolebinding "${OPENSHIFT_NAMESPACE}-hypershell-controller-scc-bind" --ignore-not-found >/dev/null 2>&1 || true + oc_cli delete clusterrolebinding "${OPENSHIFT_NAMESPACE}-hypershell-controller" --ignore-not-found >/dev/null 2>&1 || true + + info "Removing namespace group ${OPENSHIFT_NAMESPACE} and ${OPENSHIFT_KEYCLOAK_NAMESPACE}" + remove_project "${OPENSHIFT_KEYCLOAK_NAMESPACE}" + remove_project "${OPENSHIFT_NAMESPACE}" + clear_all_openshift_swaps + success "Environment ${OPENSHIFT_NAMESPACE} (and ${OPENSHIFT_KEYCLOAK_NAMESPACE}) removed" +} + +# OpenShift has no cluster to destroy. Keep the Kind-shaped target as an alias. +cluster_teardown() { + cluster_down +} + +cluster_status() { + header "OpenShift" + if ! command -v oc >/dev/null 2>&1 || ! oc_cli whoami >/dev/null 2>&1; then + warn "No reachable OpenShift cluster in the current kubeconfig context" + return 0 + fi + info "User: $(oc_cli whoami 2>/dev/null || echo unknown)" + info "Server: $(oc_cli whoami --show-server 2>/dev/null || echo unknown)" + echo "" + + resolve_openshift_namespace + validate_namespace_group + + header "Namespaces" + oc_cli get namespace "${OPENSHIFT_NAMESPACE}" "${OPENSHIFT_KEYCLOAK_NAMESPACE}" 2>/dev/null \ + || warn "Namespace group not found" + echo "" + + header "Pods (${OPENSHIFT_NAMESPACE})" + oc_cli get pods -n "${OPENSHIFT_NAMESPACE}" -o wide 2>/dev/null || warn "Namespace not found" + echo "" + header "Pods (${OPENSHIFT_KEYCLOAK_NAMESPACE})" + oc_cli get pods -n "${OPENSHIFT_KEYCLOAK_NAMESPACE}" -o wide 2>/dev/null || warn "Namespace not found" + echo "" + + header "Services" + oc_cli get svc -n "${OPENSHIFT_NAMESPACE}" 2>/dev/null || true + oc_cli get svc -n "${OPENSHIFT_KEYCLOAK_NAMESPACE}" 2>/dev/null || true + echo "" + + header "Routes" + oc_cli get route -n "${OPENSHIFT_NAMESPACE}" 2>/dev/null || true + oc_cli get route -n "${OPENSHIFT_KEYCLOAK_NAMESPACE}" 2>/dev/null || true + echo "" + + header "Gateway" + oc_cli get gateway "${GATEWAY_API_GATEWAY_NAME}" -n "${GATEWAY_API_GATEWAY_NAMESPACE}" 2>/dev/null \ + || warn "Shared Gateway ${GATEWAY_API_GATEWAY_NAMESPACE}/${GATEWAY_API_GATEWAY_NAME} not found" + oc_cli get grpcroutes -n "${OPENSHIFT_NAMESPACE}" 2>/dev/null || true + echo "" + + header "Component Swap Status" + local file + file="$(openshift_swap_file)" + if [[ -f "${file}" ]] && [[ -s "${file}" ]]; then + info "Swapped components:" + local comp image + while IFS=$'\t' read -r comp image; do + [[ -n "${comp}" ]] || continue + info " - ${comp} (working-tree ${image})" + done < "${file}" + info "Baseline components:" + for comp in api-server control-plane web-console; do + if ! is_openshift_swapped "${comp}"; then + info " - ${comp} (registry image)" + fi + done + else + info "All components running baseline (registry) images" + fi +} + +login_internal_registry() { + local registry + if [[ -n "${OPENSHIFT_IMAGE_REGISTRY:-}" ]]; then + registry="${OPENSHIFT_IMAGE_REGISTRY}" + else + registry="$(oc_cli registry info 2>/dev/null || true)" + fi + if [[ -z "${registry}" ]]; then + error "Could not discover the OpenShift internal registry." + error "Set OPENSHIFT_IMAGE_REGISTRY to a registry the cluster can pull from, then retry." + exit 1 + fi + OPENSHIFT_PUSH_REGISTRY="${registry}" + OPENSHIFT_PULL_REGISTRY="$(oc_cli registry info --internal 2>/dev/null || printf '%s' "${registry}")" + info "Logging in to registry ${OPENSHIFT_PUSH_REGISTRY}..." + if oc_cli registry login --registry "${OPENSHIFT_PUSH_REGISTRY}" >/dev/null 2>&1; then + return 0 + fi + local user token + user="$(oc_cli whoami)" + token="$(oc_cli whoami -t)" + if ${CONTAINER_ENGINE} login -u "${user}" -p "${token}" "${OPENSHIFT_PUSH_REGISTRY}" >/dev/null; then + return 0 + fi + error "Failed to log in to ${OPENSHIFT_PUSH_REGISTRY}. Set OPENSHIFT_IMAGE_REGISTRY or fix oc registry login." + exit 1 +} + +push_component_image() { + local component="$1" + component_spec "${component}" + local commit + commit="$(git -C "${REPO_ROOT}" rev-parse --short HEAD 2>/dev/null || echo unknown)" + local tag="${commit}-${OPENSHIFT_NAMESPACE}" + tag="$(printf '%s' "${tag}" | tr '[:upper:]' '[:lower:]' | sed -E 's/[^a-z0-9._-]+/-/g')" + local imagestream="hypershell-${component}" + oc_cli create imagestream "${imagestream}" -n "${OPENSHIFT_NAMESPACE}" --dry-run=client -o yaml \ + | oc_cli apply -f - >/dev/null + + local push_ref="${OPENSHIFT_PUSH_REGISTRY}/${OPENSHIFT_NAMESPACE}/${imagestream}:${tag}" + info "Building ${component} from working tree..." + ${CONTAINER_ENGINE} build -t "${LOCAL_IMAGE}" \ + -f "${REPO_ROOT}/${DOCKERFILE}" ${BUILD_ARGS[@]+"${BUILD_ARGS[@]}"} \ + "${REPO_ROOT}/${BUILD_CONTEXT}" + ${CONTAINER_ENGINE} tag "${LOCAL_IMAGE}" "${push_ref}" + info "Pushing ${push_ref}..." + ${CONTAINER_ENGINE} push "${push_ref}" + + local digest="" + digest="$(oc_cli get imagestreamtag "${imagestream}:${tag}" -n "${OPENSHIFT_NAMESPACE}" \ + -o jsonpath='{.image.metadata.name}' 2>/dev/null || true)" + local pull_ref + if [[ -n "${digest}" ]]; then + pull_ref="${OPENSHIFT_PULL_REGISTRY}/${OPENSHIFT_NAMESPACE}/${imagestream}@${digest}" + else + pull_ref="${OPENSHIFT_PULL_REGISTRY}/${OPENSHIFT_NAMESPACE}/${imagestream}:${tag}" + fi + OPENSHIFT_SWAPPED_IMAGE="${pull_ref}" +} + +rollout_component_image() { + local component="$1" + local image="$2" + component_spec "${component}" + local args=() c + for c in ${CONTAINERS}; do + args+=("${c}=${image}") + done + oc_cli set image "deployment/${DEPLOYMENT}" "${args[@]}" -n "${OPENSHIFT_NAMESPACE}" + local desired + desired="$(oc_cli get deployment "${DEPLOYMENT}" -n "${OPENSHIFT_NAMESPACE}" \ + -o jsonpath='{.spec.replicas}' 2>/dev/null || echo 0)" + if [[ "${desired:-0}" -lt 1 ]]; then + oc_cli scale "deployment/${DEPLOYMENT}" -n "${OPENSHIFT_NAMESPACE}" --replicas=1 + fi + oc_cli rollout restart "deployment/${DEPLOYMENT}" -n "${OPENSHIFT_NAMESPACE}" + oc_cli rollout status "deployment/${DEPLOYMENT}" -n "${OPENSHIFT_NAMESPACE}" --timeout=180s +} + +component_swap() { + local component="$1" + header "Swap ${component} (up)" + require_openshift_cluster + resolve_openshift_namespace + validate_namespace_group + if ! namespace_exists "${OPENSHIFT_NAMESPACE}"; then + error "No OpenShift environment in namespace '${OPENSHIFT_NAMESPACE}'. Run 'make openshift-up' first." + exit 1 + fi + refuse_foreign_namespace "${OPENSHIFT_NAMESPACE}" >/dev/null + login_internal_registry + push_component_image "${component}" + info "Rolling out ${component} to ${OPENSHIFT_SWAPPED_IMAGE}" + rollout_component_image "${component}" "${OPENSHIFT_SWAPPED_IMAGE}" + track_openshift_swap "${component}" "${OPENSHIFT_SWAPPED_IMAGE}" + success "${component} swapped to working-tree image ${OPENSHIFT_SWAPPED_IMAGE}" +} + +component_revert() { + local component="$1" + header "Swap ${component} (down)" + require_openshift_cluster + resolve_openshift_namespace + validate_namespace_group + if ! is_openshift_swapped "${component}"; then + warn "${component} is already running the baseline image." + return 0 + fi + component_spec "${component}" + info "Reverting ${component} to baseline image ${BASELINE_IMAGE}..." + rollout_component_image "${component}" "${BASELINE_IMAGE}" + clear_openshift_swap "${component}" + success "${component} reverted to baseline." +} diff --git a/scripts/cluster/lib.sh b/scripts/cluster/lib.sh new file mode 100755 index 00000000..58668fc1 --- /dev/null +++ b/scripts/cluster/lib.sh @@ -0,0 +1,255 @@ +#!/usr/bin/env bash +# Shared seams for the cluster lifecycle driver model. +# Kind and OpenShift drivers implement cluster_up, cluster_down, cluster_teardown, +# cluster_status, component_swap, and component_revert. +set -euo pipefail + +if [[ -z "${NO_COLOR:-}" ]] && [[ -t 1 ]]; then + BOLD='\033[1m' + BLUE='\033[0;34m' + CYAN='\033[0;36m' + GREEN='\033[0;32m' + YELLOW='\033[1;33m' + RED='\033[0;31m' + NC='\033[0m' +else + BOLD='' BLUE='' CYAN='' GREEN='' YELLOW='' RED='' NC='' +fi + +header() { printf "${BOLD}${BLUE}==> %s${NC}\n" "$*"; } +info() { printf "${CYAN} %s${NC}\n" "$*"; } +success() { printf "${GREEN} %s${NC}\n" "$*"; } +warn() { printf "${YELLOW} %s${NC}\n" "$*"; } +error() { printf "${RED}ERROR: %s${NC}\n" "$*" >&2; } + +CLUSTER_SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO_ROOT="$(cd "${CLUSTER_SCRIPT_DIR}/../.." && pwd)" + +: "${CONTAINER_ENGINE:=$(command -v podman 2>/dev/null || echo docker)}" +: "${IMAGE_REGISTRY:=quay.io/redhat-services-prod/hcm-eng-prod-tenant/hypershell-main}" +: "${IMAGE_TAG:=latest}" +: "${api_server_ref:=${IMAGE_REGISTRY}/hypershell-api-server-main:${IMAGE_TAG}}" +: "${control_plane_ref:=${IMAGE_REGISTRY}/hypershell-control-plane-main:${IMAGE_TAG}}" +: "${web_console_ref:=${IMAGE_REGISTRY}/hypershell-web-console-main:${IMAGE_TAG}}" +: "${api_server_local:=localhost/hypershell:dev}" +: "${control_plane_local:=localhost/hypershell-controller:dev}" +: "${web_console_local:=localhost/hypershell-web-console:dev}" +: "${build_version:=$(git -C "${REPO_ROOT}" rev-parse --short HEAD 2>/dev/null || echo unknown)}" +: "${build_time:=$(date -u '+%Y-%m-%d %H:%M:%S UTC')}" +: "${GATEWAY_IMAGE:=ghcr.io/nvidia/openshell/gateway:0.0.109}" +: "${GATEWAY_API_GATEWAY_NAME:=openshell-grpc-gateway}" +: "${GATEWAY_API_GATEWAY_NAMESPACE:=openshift-ingress}" + +# RFC 1123 DNS label: [a-z0-9]([a-z0-9-]*[a-z0-9])? up to 63 characters. +# OpenShift platform namespaces are capped at 54 so "${ns}-keycloak" stays <= 63. +RFC1123_LABEL_RE='^[a-z0-9]([-a-z0-9]*[a-z0-9])?$' + +validate_rfc1123_label() { + local name="$1" + local max="${2:-63}" + if [[ -z "${name}" ]]; then + error "Name is empty (must be an RFC 1123 DNS label of at most ${max} characters)" + return 1 + fi + if (( ${#name} > max )); then + error "Name '${name}' is ${#name} characters; maximum is ${max}" + return 1 + fi + if [[ ! "${name}" =~ ${RFC1123_LABEL_RE} ]]; then + error "Name '${name}' is not a valid RFC 1123 DNS label (lowercase alphanumeric and '-', must start and end with alphanumeric)" + return 1 + fi +} + +# Map an identity (oc whoami, email, kube:admin) to an RFC 1123 DNS label. +sanitize_dns_label() { + local raw="$1" + local max="${2:-54}" + local out + out=$(printf '%s' "${raw}" | tr '[:upper:]' '[:lower:]' | sed -E 's/[^a-z0-9-]+/-/g; s/-+/-/g; s/^-+//; s/-+$//') + if [[ -z "${out}" ]]; then + printf '' + return 1 + fi + if [[ ! "${out:0:1}" =~ [a-z0-9] ]]; then + out="ns-${out}" + fi + if (( ${#out} > max )); then + out="${out:0:${max}}" + out="${out%-}" + fi + if [[ ! "${out: -1}" =~ [a-z0-9] ]]; then + out="${out%-}x" + fi + printf '%s' "${out}" +} + +keycloak_namespace_for() { + printf '%s-keycloak' "$1" +} + +# Cluster-scoped / system namespaces that local-dev must never claim. +is_reserved_cluster_namespace() { + case "$1" in + default|openshift|kube-system|kube-public|kube-node-lease) return 0 ;; + kube-*|openshift-*) return 0 ;; + esac + return 1 +} + +# Strip a Gateway listener hostname (e.g. *.openshell.example.com) down to the +# tenant base domain the control plane expects (openshell.example.com). +gateway_base_domain_from_hostname() { + local host="$1" + host="${host#\*.}" + printf '%s' "${host}" +} + +load_cluster_driver() { + local driver="${CLUSTER_DRIVER:-}" + if [[ -z "${driver}" ]]; then + error "CLUSTER_DRIVER is not set. Use 'make kind-up' or 'make openshift-up'." + return 1 + fi + local path="${CLUSTER_SCRIPT_DIR}/drivers/${driver}.sh" + if [[ ! -f "${path}" ]]; then + error "Unknown cluster driver '${driver}'. Expected ${path}" + error "Available drivers: kind, openshift" + return 1 + fi + # shellcheck source=/dev/null + source "${path}" + local fn + for fn in cluster_up cluster_down cluster_teardown cluster_status component_swap component_revert; do + if ! declare -F "${fn}" >/dev/null; then + error "Driver '${driver}' does not implement ${fn}" + return 1 + fi + done +} + +# Component lookup used by both drivers' swap paths. +# Sets: DEPLOYMENT CONTAINERS LOCAL_IMAGE BASELINE_IMAGE DOCKERFILE BUILD_CONTEXT BUILD_ARGS +component_spec() { + local component="$1" + BUILD_ARGS=() + case "${component}" in + api-server) + DEPLOYMENT="hypershell-api-server" + CONTAINERS="api-server migrate" + LOCAL_IMAGE="${api_server_local}" + BASELINE_IMAGE="${api_server_ref}" + DOCKERFILE="components/api-server/Dockerfile" + BUILD_CONTEXT="components/api-server" + BUILD_ARGS=(--build-arg "GIT_VERSION=${build_version}" --build-arg "BUILD_TIME=${build_time}") + ;; + control-plane) + DEPLOYMENT="hypershell-controller" + CONTAINERS="controller" + LOCAL_IMAGE="${control_plane_local}" + BASELINE_IMAGE="${control_plane_ref}" + DOCKERFILE="components/control-plane/Dockerfile" + BUILD_CONTEXT="." + ;; + web-console) + DEPLOYMENT="hypershell-web-console" + CONTAINERS="web-console" + LOCAL_IMAGE="${web_console_local}" + BASELINE_IMAGE="${web_console_ref}" + DOCKERFILE="components/web-console/Dockerfile" + BUILD_CONTEXT="." + ;; + *) + error "Unknown component: ${component}" + error "Valid components: api-server, control-plane, web-console" + return 1 + ;; + esac +} + +# Per-namespace OpenShift swap ledger. Kind keeps using scripts/kind/.kind-swaps. +openshift_swap_dir() { + printf '%s/.openshift-swaps' "${REPO_ROOT}" +} + +openshift_swap_file() { + local ns="${1:-${OPENSHIFT_NAMESPACE:-}}" + printf '%s/%s' "$(openshift_swap_dir)" "${ns}" +} + +track_openshift_swap() { + local component="$1" + local image="$2" + local file + file="$(openshift_swap_file)" + mkdir -p "$(openshift_swap_dir)" + touch "${file}" + if grep -q "^${component}[[:space:]]" "${file}" 2>/dev/null; then + local tmp + tmp="$(mktemp)" + sed "/^${component}[[:space:]]/d" "${file}" > "${tmp}" + mv "${tmp}" "${file}" + fi + printf '%s\t%s\n' "${component}" "${image}" >> "${file}" +} + +clear_openshift_swap() { + local component="$1" + local file + file="$(openshift_swap_file)" + if [[ -f "${file}" ]]; then + local tmp + tmp="$(mktemp)" + sed "/^${component}[[:space:]]/d" "${file}" > "${tmp}" + mv "${tmp}" "${file}" + if [[ ! -s "${file}" ]]; then + rm -f "${file}" + fi + fi +} + +is_openshift_swapped() { + local component="$1" + local file + file="$(openshift_swap_file)" + [[ -f "${file}" ]] && grep -q "^${component}[[:space:]]" "${file}" +} + +openshift_swap_image() { + local component="$1" + local file + file="$(openshift_swap_file)" + [[ -f "${file}" ]] || return 0 + awk -F '\t' -v c="${component}" '$1 == c { print $2; exit }' "${file}" +} + +clear_all_openshift_swaps() { + rm -f "$(openshift_swap_file)" +} + +json_string_field() { + python3 -c 'import json,sys +doc=json.load(sys.stdin) +val=doc.get(sys.argv[1],"") +print("" if val is None else val) +' "$1" +} + +json_first_id() { + python3 -c 'import json,sys +docs=json.load(sys.stdin) +print(docs[0]["id"] if docs else "") +' +} + +# Restrict a Keycloak client representation to this console origin. +# Spec: oidc-integration Identity Provider Client Security — no wildcards. +keycloak_client_with_console_redirects() { + local console_host="$1" + python3 -c 'import json,sys +host=sys.argv[1] +doc=json.load(sys.stdin) +doc["redirectUris"]=[f"https://{host}/auth/callback", f"https://{host}"] +json.dump(doc, sys.stdout) +' "${console_host}" +} diff --git a/scripts/cluster/lib_test.sh b/scripts/cluster/lib_test.sh new file mode 100755 index 00000000..2ab9d53e --- /dev/null +++ b/scripts/cluster/lib_test.sh @@ -0,0 +1,464 @@ +#!/usr/bin/env bash +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# shellcheck source=lib.sh +source "${SCRIPT_DIR}/lib.sh" + +PASS=0 +FAIL=0 + +assert_eq() { + local want="$1" got="$2" label="$3" + if [[ "${want}" == "${got}" ]]; then + PASS=$((PASS + 1)) + else + FAIL=$((FAIL + 1)) + printf 'FAIL: %s (want=%q got=%q)\n' "${label}" "${want}" "${got}" + fi +} + +assert_ok() { + local label="$1" + shift + if "$@" >/dev/null 2>&1; then + PASS=$((PASS + 1)) + else + FAIL=$((FAIL + 1)) + printf 'FAIL: %s (expected success)\n' "${label}" + fi +} + +assert_fail() { + local label="$1" + shift + if "$@" >/dev/null 2>&1; then + FAIL=$((FAIL + 1)) + printf 'FAIL: %s (expected failure)\n' "${label}" + else + PASS=$((PASS + 1)) + fi +} + +# --- RFC 1123 / sanitization --- +assert_eq "kube-admin" "$(sanitize_dns_label 'kube:admin')" "sanitize kube:admin" +assert_eq "user-redhat-com" "$(sanitize_dns_label 'user@redhat.com')" "sanitize email" +assert_eq "alice" "$(sanitize_dns_label 'Alice')" "sanitize mixed case" +assert_ok "valid short label" validate_rfc1123_label "alice" 54 +assert_ok "54-char label" validate_rfc1123_label "$(printf 'a%.0s' {1..54})" 54 +assert_fail "55-char platform label" validate_rfc1123_label "$(printf 'a%.0s' {1..55})" 54 +assert_fail "uppercase rejected" validate_rfc1123_label "Alice" 54 +assert_fail "underscore rejected" validate_rfc1123_label "alice_dev" 54 +assert_eq "tok" "$(printf '%s' '{"access_token":"tok","expires_in":60}' | json_string_field access_token)" "json_string_field access_token" +assert_eq "abc-id" "$(printf '%s' '[{"id":"abc-id","clientId":"hypershell-frontend"}]' | json_first_id)" "json_first_id" +merged="$(printf '%s' '{"id":"x","redirectUris":["https://console.hypershell.localhost/*"]}' | keycloak_client_with_console_redirects 'console.apps.example.com')" +assert_eq '["https://console.apps.example.com/auth/callback", "https://console.apps.example.com"]' \ + "$(printf '%s' "${merged}" | python3 -c 'import json,sys; print(json.dumps(json.load(sys.stdin)["redirectUris"]))')" \ + "keycloak_client_with_console_redirects replaces Kind localhost URIs" +if grep -A3 '^cluster_teardown()' "${SCRIPT_DIR}/drivers/openshift.sh" | grep -q 'cluster_down'; then + PASS=$((PASS + 1)) +else + FAIL=$((FAIL + 1)) + echo 'FAIL: OpenShift cluster_teardown is not an alias of cluster_down' +fi +if grep -A25 '^remove_project()' "${SCRIPT_DIR}/drivers/openshift.sh" | grep -q -- '--wait=false'; then + FAIL=$((FAIL + 1)) + echo 'FAIL: remove_project deletes projects without waiting' +else + PASS=$((PASS + 1)) +fi +if grep -E 'deletion started|removal started' "${SCRIPT_DIR}/drivers/openshift.sh" >/dev/null; then + FAIL=$((FAIL + 1)) + echo 'FAIL: OpenShift down still reports deletion as only started' +else + PASS=$((PASS + 1)) +fi +if grep -A40 '^cluster_down()' "${SCRIPT_DIR}/drivers/openshift.sh" | grep -q 'OPENSHIFT_KEYCLOAK_NAMESPACE'; then + PASS=$((PASS + 1)) +else + FAIL=$((FAIL + 1)) + echo 'FAIL: OpenShift cluster_down does not remove the Keycloak namespace' +fi +if grep -E 'oc_cli exec.*curl|oc exec.*curl' "${SCRIPT_DIR}/drivers/openshift.sh" >/dev/null; then + FAIL=$((FAIL + 1)) + echo 'FAIL: OpenShift driver still oc execs curl in a pod' +else + PASS=$((PASS + 1)) +fi +assert_eq "alice-keycloak" "$(keycloak_namespace_for alice)" "keycloak namespace suffix" +assert_ok "derived keycloak ns fits 63" validate_rfc1123_label "$(keycloak_namespace_for "$(printf 'a%.0s' {1..54})")" 63 +assert_eq "openshell.example.com" "$(gateway_base_domain_from_hostname '*.openshell.example.com')" "strip wildcard listener hostname" +assert_eq "openshell.example.com" "$(gateway_base_domain_from_hostname 'openshell.example.com')" "passthrough non-wildcard hostname" +assert_fail "alice is not reserved" is_reserved_cluster_namespace alice +assert_fail "hypershell-e2e-test is not reserved" is_reserved_cluster_namespace hypershell-e2e-test +assert_ok "default is reserved" is_reserved_cluster_namespace default +assert_ok "openshift-ingress is reserved" is_reserved_cluster_namespace openshift-ingress +assert_ok "kube-system is reserved" is_reserved_cluster_namespace kube-system + +# --- Driver loading --- +CLUSTER_DRIVER=kind +assert_ok "kind driver loads" load_cluster_driver +declare -F cluster_up >/dev/null && PASS=$((PASS + 1)) || { FAIL=$((FAIL + 1)); echo 'FAIL: kind cluster_up missing'; } + +CLUSTER_DRIVER=openshift +# Re-source a fresh shell function table by loading again in a subshell +assert_ok "openshift driver loads" bash -c ' + source "'"${SCRIPT_DIR}"'/lib.sh" + CLUSTER_DRIVER=openshift + load_cluster_driver + declare -F cluster_up cluster_down cluster_teardown cluster_status component_swap component_revert >/dev/null +' + +CLUSTER_DRIVER=not-a-driver +assert_fail "unknown driver rejected" load_cluster_driver + +# --- Swap ledger --- +OPENSHIFT_NAMESPACE=test-alice +swap_root="$(mktemp -d)" +# Point the ledger at a temp dir by overriding REPO_ROOT via a subshell helper +ledger_test() { + local tmp="$1" + ( + # shellcheck source=lib.sh + source "${SCRIPT_DIR}/lib.sh" + REPO_ROOT="${tmp}" + OPENSHIFT_NAMESPACE=test-alice + track_openshift_swap api-server "image.example/test-alice/hypershell-api-server@sha256:abc" + is_openshift_swapped api-server + [[ "$(openshift_swap_image api-server)" == "image.example/test-alice/hypershell-api-server@sha256:abc" ]] + ! is_openshift_swapped control-plane + track_openshift_swap api-server "image.example/test-alice/hypershell-api-server@sha256:def" + [[ "$(openshift_swap_image api-server)" == "image.example/test-alice/hypershell-api-server@sha256:def" ]] + clear_openshift_swap api-server + ! is_openshift_swapped api-server + ) +} +assert_ok "openshift swap ledger" ledger_test "${swap_root}" +rm -rf "${swap_root}" + +# --- Namespace rewriter --- +fixture="$(mktemp)" +cat > "${fixture}" <<'EOF' +apiVersion: v1 +kind: Namespace +metadata: + name: hypershell-system +--- +apiVersion: v1 +kind: Namespace +metadata: + name: keycloak +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: keycloak + namespace: keycloak +spec: + template: + spec: + containers: + - name: keycloak + env: + - name: KC_HOSTNAME + value: https://keycloak.hypershell.localhost +--- +apiVersion: v1 +kind: Service +metadata: + name: keycloak-service + namespace: keycloak +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: hypershell-controller + namespace: hypershell-system +spec: + template: + spec: + containers: + - name: controller + env: + - name: HYPERSHELL_NAMESPACE + value: hypershell-system +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: hypershell-controller +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: hypershell-controller +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: hypershell-controller +subjects: + - kind: ServiceAccount + name: hypershell-controller + namespace: hypershell-system +EOF + +rewritten="$(python3 "${SCRIPT_DIR}/rewrite-namespaces.py" \ + --platform-namespace alice \ + --keycloak-namespace alice-keycloak < "${fixture}")" +rm -f "${fixture}" + +assert_eq 0 "$(printf '%s' "${rewritten}" | grep -c hypershell-system || true)" "no leftover hypershell-system" +printf '%s' "${rewritten}" | grep -q $'kind: Namespace\nmetadata:\n name: alice$' \ + || printf '%s' "${rewritten}" | grep -A2 'kind: Namespace' | grep -q 'name: alice' +if printf '%s' "${rewritten}" | grep -q 'name: alice$'; then + PASS=$((PASS + 1)) +else + FAIL=$((FAIL + 1)) + echo 'FAIL: platform namespace name not rewritten to alice' +fi +if printf '%s' "${rewritten}" | grep -q 'name: alice-keycloak'; then + PASS=$((PASS + 1)) +else + FAIL=$((FAIL + 1)) + echo 'FAIL: keycloak namespace name not rewritten' +fi +if printf '%s' "${rewritten}" | grep -q 'namespace: alice-keycloak'; then + PASS=$((PASS + 1)) +else + FAIL=$((FAIL + 1)) + echo 'FAIL: keycloak resource namespace not rewritten' +fi +if printf '%s' "${rewritten}" | grep -q 'name: keycloak-service'; then + PASS=$((PASS + 1)) +else + FAIL=$((FAIL + 1)) + echo 'FAIL: keycloak-service name was rewritten' +fi +if printf '%s' "${rewritten}" | grep -q 'name: alice-hypershell-controller$'; then + PASS=$((PASS + 1)) +else + FAIL=$((FAIL + 1)) + echo 'FAIL: ClusterRoleBinding name not prefixed' +fi +if printf '%s' "${rewritten}" | grep -q 'controllerroleRef'; then + FAIL=$((FAIL + 1)) + echo 'FAIL: ClusterRoleBinding name was glued onto roleRef' +else + PASS=$((PASS + 1)) +fi +# ClusterRole name and roleRef.name must stay unprefixed +if printf '%s' "${rewritten}" | grep -q 'kind: ClusterRole' \ + && printf '%s' "${rewritten}" | awk '/kind: ClusterRole$/{p=1} p&&/name:/{print; exit}' | grep -q 'name: hypershell-controller'; then + PASS=$((PASS + 1)) +else + FAIL=$((FAIL + 1)) + echo 'FAIL: ClusterRole name should stay hypershell-controller' +fi +if printf '%s' "${rewritten}" | grep -A4 'roleRef:' | grep -q 'name: hypershell-controller'; then + PASS=$((PASS + 1)) +else + FAIL=$((FAIL + 1)) + echo 'FAIL: roleRef.name should stay hypershell-controller' +fi +if printf '%s' "${rewritten}" | grep -q 'value: alice'; then + PASS=$((PASS + 1)) +else + FAIL=$((FAIL + 1)) + echo 'FAIL: HYPERSHELL_NAMESPACE value not rewritten' +fi + +omit_fixture="$(mktemp)" +cat > "${omit_fixture}" <<'EOF' +apiVersion: v1 +kind: Namespace +metadata: + name: hypershell-system +--- +apiVersion: v1 +kind: Service +metadata: + name: api + namespace: hypershell-system +EOF +omitted="$(python3 "${SCRIPT_DIR}/rewrite-namespaces.py" \ + --platform-namespace alice \ + --keycloak-namespace alice-keycloak \ + --omit-namespaces < "${omit_fixture}")" +rm -f "${omit_fixture}" +if printf '%s' "${omitted}" | grep -q 'kind: Namespace'; then + FAIL=$((FAIL + 1)) + echo 'FAIL: --omit-namespaces still contains Namespace' +else + PASS=$((PASS + 1)) +fi +if printf '%s' "${omitted}" | grep -q 'kind: Service'; then + PASS=$((PASS + 1)) +else + FAIL=$((FAIL + 1)) + echo 'FAIL: --omit-namespaces dropped non-Namespace resources' +fi + +only_kc="$(python3 "${SCRIPT_DIR}/rewrite-namespaces.py" \ + --platform-namespace alice \ + --keycloak-namespace alice-keycloak \ + --omit-namespaces \ + --only-namespace alice-keycloak <<<"${rewritten}")" +if printf '%s' "${only_kc}" | grep -q 'namespace: alice-keycloak'; then + PASS=$((PASS + 1)) +else + FAIL=$((FAIL + 1)) + echo 'FAIL: --only-namespace alice-keycloak dropped Keycloak resources' +fi +if printf '%s' "${only_kc}" | grep -q 'namespace: alice$'; then + FAIL=$((FAIL + 1)) + echo 'FAIL: --only-namespace alice-keycloak kept platform resources' +else + PASS=$((PASS + 1)) +fi +if printf '%s' "${only_kc}" | grep -q 'kind: ClusterRole'; then + FAIL=$((FAIL + 1)) + echo 'FAIL: --only-namespace alice-keycloak kept cluster-scoped resources' +else + PASS=$((PASS + 1)) +fi + +only_plat="$(python3 "${SCRIPT_DIR}/rewrite-namespaces.py" \ + --platform-namespace alice \ + --keycloak-namespace alice-keycloak \ + --omit-namespaces \ + --only-namespace alice \ + --include-cluster-scoped <<<"${rewritten}")" +if printf '%s' "${only_plat}" | grep -q 'kind: ClusterRole'; then + PASS=$((PASS + 1)) +else + FAIL=$((FAIL + 1)) + echo 'FAIL: --include-cluster-scoped dropped ClusterRole' +fi +# ClusterRoleBinding prefix must not glue metadata.name onto roleRef +crb_fixture="$(mktemp)" +cat > "${crb_fixture}" <<'EOF' +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + labels: + app.kubernetes.io/name: hypershell + name: hypershell-controller +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: hypershell-controller +subjects: +- kind: ServiceAccount + name: hypershell-controller + namespace: hypershell-system +EOF +crb_out="$(python3 "${SCRIPT_DIR}/rewrite-namespaces.py" \ + --platform-namespace alice \ + --keycloak-namespace alice-keycloak \ + --omit-namespaces \ + --only-namespace __cluster__ \ + --include-cluster-scoped < "${crb_fixture}")" +rm -f "${crb_fixture}" +if printf '%s' "${crb_out}" | grep -qx ' name: alice-hypershell-controller'; then + PASS=$((PASS + 1)) +else + FAIL=$((FAIL + 1)) + echo 'FAIL: kustomize-style ClusterRoleBinding metadata.name not prefixed on its own line' + printf '%s\n' "${crb_out}" +fi +if printf '%s' "${crb_out}" | grep -qx 'roleRef:'; then + PASS=$((PASS + 1)) +else + FAIL=$((FAIL + 1)) + echo 'FAIL: roleRef was not kept as its own line' +fi +omit_cr="$(python3 "${SCRIPT_DIR}/rewrite-namespaces.py" \ + --platform-namespace alice \ + --keycloak-namespace alice-keycloak \ + --omit-namespaces \ + --omit-kinds ClusterRole,ClusterRoleBinding \ + --only-namespace alice \ + --include-cluster-scoped <<<"${rewritten}")" +if printf '%s' "${omit_cr}" | grep -q 'kind: ClusterRole'; then + FAIL=$((FAIL + 1)) + echo 'FAIL: --omit-kinds kept ClusterRole' +else + PASS=$((PASS + 1)) +fi + +stripped="$(python3 "${SCRIPT_DIR}/rewrite-namespaces.py" \ + --platform-namespace alice \ + --keycloak-namespace alice-keycloak \ + --omit-namespaces \ + --strip-openshift-uids \ + < "${REPO_ROOT}/deploy/base/postgres.yaml")" +if printf '%s' "${stripped}" | grep -q 'runAsUser\|fsGroup'; then + FAIL=$((FAIL + 1)) + echo 'FAIL: --strip-openshift-uids left runAsUser or fsGroup' +else + PASS=$((PASS + 1)) +fi +if printf '%s' "${stripped}" | grep -q 'runAsNonRoot: true'; then + PASS=$((PASS + 1)) +else + FAIL=$((FAIL + 1)) + echo 'FAIL: --strip-openshift-uids dropped runAsNonRoot' +fi + +# --- Overlay still renders --- +if command -v kustomize >/dev/null 2>&1; then + if kustomize build --load-restrictor=LoadRestrictionsNone "${REPO_ROOT}/deploy/openshift" >/dev/null; then + PASS=$((PASS + 1)) + else + FAIL=$((FAIL + 1)) + echo 'FAIL: kustomize build deploy/openshift' + fi + if kustomize build "${REPO_ROOT}/deploy/hub" >/dev/null; then + PASS=$((PASS + 1)) + else + FAIL=$((FAIL + 1)) + echo 'FAIL: kustomize build deploy/hub' + fi + os_out="$(mktemp)" + if kustomize build --load-restrictor=LoadRestrictionsNone "${REPO_ROOT}/deploy/openshift" \ + | python3 "${SCRIPT_DIR}/rewrite-namespaces.py" \ + --platform-namespace alice \ + --keycloak-namespace alice-keycloak \ + > "${os_out}"; then + if grep -q 'kind: Route' "${os_out}"; then + PASS=$((PASS + 1)) + else + FAIL=$((FAIL + 1)) + echo 'FAIL: rewritten overlay missing Route' + fi + if grep -q 'name: alice-keycloak' "${os_out}"; then + PASS=$((PASS + 1)) + else + FAIL=$((FAIL + 1)) + echo 'FAIL: rewritten overlay missing alice-keycloak namespace' + fi + kc_np="$(awk '/name: keycloak-allow-platform/,/^---$/' "${os_out}")" + if printf '%s' "${kc_np}" | grep -q 'namespace: alice-keycloak' \ + && printf '%s' "${kc_np}" | grep -q 'kubernetes.io/metadata.name: alice'; then + PASS=$((PASS + 1)) + else + FAIL=$((FAIL + 1)) + echo 'FAIL: rewritten overlay missing Keycloak allow-from-platform NetworkPolicy' + fi + if ! grep -q hypershell-system "${os_out}"; then + PASS=$((PASS + 1)) + else + FAIL=$((FAIL + 1)) + echo 'FAIL: rewritten overlay still contains hypershell-system' + fi + else + FAIL=$((FAIL + 1)) + echo 'FAIL: rewritten overlay render' + fi + rm -f "${os_out}" +else + warn "kustomize not installed; skipping overlay render checks" +fi + +printf 'OpenShift lifecycle tests: %d passed, %d failed\n' "${PASS}" "${FAIL}" +[[ "${FAIL}" -eq 0 ]] diff --git a/scripts/cluster/rewrite-namespaces.py b/scripts/cluster/rewrite-namespaces.py new file mode 100755 index 00000000..e40ba0fe --- /dev/null +++ b/scripts/cluster/rewrite-namespaces.py @@ -0,0 +1,214 @@ +#!/usr/bin/env python3 +"""Rewrite kustomize-rendered YAML for an ephemeral OpenShift namespace group. + +Maps the overlay's platform namespace (hypershell-system) to OPENSHIFT_NAMESPACE +and the bundled Keycloak namespace (keycloak) to ${OPENSHIFT_NAMESPACE}-keycloak. +ClusterRoleBindings are renamed so two environments on one cluster do not share +a binding name. ClusterRoles stay shared. +""" +from __future__ import annotations + +import argparse +import re +import sys + +PLATFORM_NS = "hypershell-system" +KEYCLOAK_NS = "keycloak" + + +def split_docs(text: str) -> list[str]: + if text.startswith("---"): + body = text[3:] + prefix_sep = True + else: + body = text + prefix_sep = False + docs = re.split(r"\n---\n", body) + if prefix_sep and docs and docs[0].strip() == "": + docs = docs[1:] + return docs + + +def kind_of(doc: str) -> str: + match = re.search(r"^kind:\s*(\S+)\s*$", doc, re.M) + return match.group(1) if match else "" + + +def prefix_cluster_role_binding_name(doc: str, prefix: str) -> str: + """Prefix metadata.name only; leave roleRef.name unchanged.""" + parts = re.split(r"^(roleRef:|subjects:)", doc, maxsplit=1, flags=re.M) + head = parts[0] + rest = "".join(parts[1:]) if len(parts) > 1 else "" + + def repl(match: re.Match[str]) -> str: + name = match.group(2) + if name.startswith(prefix): + return match.group(0) + return f"{match.group(1)}{prefix}{name}" + + # Do not use \\s*$: $ is end-of-line, but \\s matches newlines and would + # swallow the newline before roleRef, gluing `name: foo` onto `roleRef:`. + head = re.sub(r"^( name:\s*)(\S+)[^\S\n]*$", repl, head, count=1, flags=re.M) + return head + rest + + +def rewrite_doc(doc: str, platform_ns: str, keycloak_ns: str) -> str: + kind = kind_of(doc) + rewritten = doc.replace(PLATFORM_NS, platform_ns) + rewritten = re.sub( + r"^(\s*namespace:\s*)" + re.escape(KEYCLOAK_NS) + r"\s*$", + rf"\g<1>{keycloak_ns}", + rewritten, + flags=re.M, + ) + if kind == "Namespace": + rewritten = re.sub( + r"^( name:\s*)" + re.escape(KEYCLOAK_NS) + r"\s*$", + rf"\g<1>{keycloak_ns}", + rewritten, + flags=re.M, + ) + if kind == "ClusterRoleBinding": + rewritten = prefix_cluster_role_binding_name(rewritten, f"{platform_ns}-") + return rewritten + + +def resource_namespace(doc: str) -> str | None: + """Return metadata.namespace, not subject or pod-template namespaces.""" + match = re.search(r"^metadata:\n((?: .*\n?)*)", doc, re.M) + if not match: + return None + ns = re.search(r"^ namespace:\s*(\S+)\s*$", match.group(1), re.M) + return ns.group(1) if ns else None + + +def metadata_name(doc: str) -> str | None: + match = re.search(r"^metadata:\n((?: .*\n?)*)", doc, re.M) + if not match: + return None + name = re.search(r"^ name:\s*(\S+)\s*$", match.group(1), re.M) + return name.group(1) if name else None + + +def keep_doc( + doc: str, + *, + omit_namespaces: bool, + only_namespace: str | None, + include_cluster_scoped: bool, + omit_kinds: set[str] | None = None, + omit_names: set[str] | None = None, +) -> bool: + if not doc.strip(): + return False + kind = kind_of(doc) + if omit_namespaces and kind == "Namespace": + return False + if omit_kinds and kind in omit_kinds: + return False + if omit_names: + name = metadata_name(doc) + if name and name in omit_names: + return False + if only_namespace is None: + return True + ns = resource_namespace(doc) + if ns is None: + return include_cluster_scoped + return ns == only_namespace + + +def rewrite( + text: str, + platform_ns: str, + keycloak_ns: str, + omit_namespaces: bool = False, + only_namespace: str | None = None, + include_cluster_scoped: bool = False, + omit_kinds: set[str] | None = None, + omit_names: set[str] | None = None, +) -> str: + docs = [rewrite_doc(doc, platform_ns, keycloak_ns) for doc in split_docs(text)] + docs = [ + doc + for doc in docs + if keep_doc( + doc, + omit_namespaces=omit_namespaces, + only_namespace=only_namespace, + include_cluster_scoped=include_cluster_scoped, + omit_kinds=omit_kinds, + omit_names=omit_names, + ) + ] + rendered = "\n---\n".join(docs) + if rendered and not rendered.endswith("\n"): + rendered += "\n" + return rendered + + +OPENSHIFT_UID_FIELDS_RE = re.compile( + r"^[ \t]+(?:runAsUser|runAsGroup|fsGroup|fsGroupChangePolicy):[^\n]*\n", + re.M, +) + + +def strip_openshift_fixed_uids(text: str) -> str: + """Drop pinned UIDs/GIDs so OpenShift restricted SCC can assign the range.""" + return OPENSHIFT_UID_FIELDS_RE.sub("", text) + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--platform-namespace", required=True) + parser.add_argument("--keycloak-namespace", required=True) + parser.add_argument( + "--omit-namespaces", + action="store_true", + help="Drop Namespace documents. Projects are created with oc new-project.", + ) + parser.add_argument( + "--only-namespace", + help="Keep only documents whose metadata.namespace matches this value.", + ) + parser.add_argument( + "--include-cluster-scoped", + action="store_true", + help="When --only-namespace is set, also keep documents with no namespace (ClusterRole, ClusterRoleBinding).", + ) + parser.add_argument( + "--omit-kinds", + default="", + help="Comma-separated kinds to drop (e.g. ClusterRole,ClusterRoleBinding,Cluster).", + ) + parser.add_argument( + "--omit-names", + default="", + help="Comma-separated metadata.names to drop (e.g. hypershell-sandbox-scc).", + ) + parser.add_argument( + "--strip-openshift-uids", + action="store_true", + help="Remove runAsUser/runAsGroup/fsGroup so restricted SCC can assign identities.", + ) + args = parser.parse_args() + omit_kinds = {k for k in args.omit_kinds.split(",") if k} + omit_names = {n for n in args.omit_names.split(",") if n} + rendered = rewrite( + sys.stdin.read(), + args.platform_namespace, + args.keycloak_namespace, + omit_namespaces=args.omit_namespaces, + only_namespace=args.only_namespace, + include_cluster_scoped=args.include_cluster_scoped, + omit_kinds=omit_kinds or None, + omit_names=omit_names or None, + ) + if args.strip_openshift_uids: + rendered = strip_openshift_fixed_uids(rendered) + sys.stdout.write(rendered) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/cluster/status.sh b/scripts/cluster/status.sh new file mode 100755 index 00000000..2821fbbe --- /dev/null +++ b/scripts/cluster/status.sh @@ -0,0 +1,8 @@ +#!/usr/bin/env bash +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# shellcheck source=lib.sh +source "${SCRIPT_DIR}/lib.sh" +load_cluster_driver +cluster_status diff --git a/scripts/cluster/swap.sh b/scripts/cluster/swap.sh new file mode 100755 index 00000000..87fa1fbc --- /dev/null +++ b/scripts/cluster/swap.sh @@ -0,0 +1,24 @@ +#!/usr/bin/env bash +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# shellcheck source=lib.sh +source "${SCRIPT_DIR}/lib.sh" + +ACTION="${1:-}" +COMPONENT="${2:-}" +if [[ -z "${ACTION}" ]] || [[ -z "${COMPONENT}" ]]; then + error "Usage: swap.sh up|down " + exit 1 +fi + +load_cluster_driver +case "${ACTION}" in + up) component_swap "${COMPONENT}" ;; + down) component_revert "${COMPONENT}" ;; + *) + error "Unknown action: ${ACTION}" + error "Valid actions: up, down" + exit 1 + ;; +esac diff --git a/scripts/cluster/teardown.sh b/scripts/cluster/teardown.sh new file mode 100755 index 00000000..ccb26000 --- /dev/null +++ b/scripts/cluster/teardown.sh @@ -0,0 +1,8 @@ +#!/usr/bin/env bash +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# shellcheck source=lib.sh +source "${SCRIPT_DIR}/lib.sh" +load_cluster_driver +cluster_teardown diff --git a/scripts/cluster/up.sh b/scripts/cluster/up.sh new file mode 100755 index 00000000..cf045629 --- /dev/null +++ b/scripts/cluster/up.sh @@ -0,0 +1,8 @@ +#!/usr/bin/env bash +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# shellcheck source=lib.sh +source "${SCRIPT_DIR}/lib.sh" +load_cluster_driver +cluster_up diff --git a/skills/RECONCILE.md b/skills/RECONCILE.md index 086ba0b6..76c72cb9 100644 --- a/skills/RECONCILE.md +++ b/skills/RECONCILE.md @@ -48,9 +48,9 @@ skills/ ## Reconciliation State -**Last analyzed**: 2026-08-31 (Keycloak event-storm KC-ES-W1 complete; rebased in e2e performance harness and OpenShift manual e2e driver features from 2026-08-25/26) +**Last analyzed**: 2026-08-31 (Keycloak event-storm KC-ES-W1 complete; OpenShift local-dev lifecycle and manual e2e driver features rebased in from 2026-08-25/27) **Spec corpus**: 40 spec files; the coverage table tracks 32 analyzed feature/spec groups after adding OpenShell Gateway Console and OpenShift Development -**Codebase commit**: working tree (Keycloak event-storm KC-ES-W1 complete) +**Codebase commit**: working tree (Keycloak event-storm KC-ES-W1 + OpenShift local-dev) ### Coverage Summary @@ -71,12 +71,12 @@ skills/ | Platform - Sandbox Count | 1 | 6 | 6 | 0 | 0 | 0 | 100% | | Platform - Local Development | 1 | 25 | 23 | 0 | 1 | 1 | 96% | | Platform - E2E Testing | 1 | 19 | 19 | 0 | 0 | 0 | 100% | -| Platform - OpenShift Development | 1 | 13 | 0 | 1 | 12 | 0 | 4% | +| Platform - OpenShift Development | 1 | 13 | 7 | 2 | 4 | 0 | 54% | | Platform - OIDC Integration | 1 | 7 | 6 | 1 | 0 | 0 | 93% | | Web Console - Architecture | 1 | 28 | 21 | 5 | 2 | 0 | 86% | | Security - RBAC Enforcement | 1 | 13 | 11 | 0 | 0 | 2 | 85% | | Standards | 13 | 0 | 0 | 0 | 0 | 0 | N/A | -| **TOTAL** | **32** | **249** | **187** | **19** | **38** | **5** | **79%** | +| **TOTAL** | **32** | **249** | **194** | **20** | **30** | **5** | **82%** | ### Spec Dependency Order @@ -120,13 +120,25 @@ Layer 7: web-console/architecture (depends on data-model, security, UI - GC-W1 added the OpenShift Route adapter and made provisioning, readiness, cleanup, and health repair use the selected ingress mode. - The base controller role permits Route CRUD and create and update access to `routes/custom-host` because the console Route sets `spec.host`. -### openshift-development.spec.md (manual e2e/performance slice) +### openshift-development.spec.md (local-dev lifecycle) | # | Requirement | Status | Gap | Code Location | Wave | |---|-------------|--------|-----|---------------|------| -| OS-E2E-1 | OpenShift E2E Driver | Partial | Manual `make e2e` and `make e2e-performance` can target an existing OpenShift environment. Lifecycle deployment, overlay namespace parameterization, infrastructure bootstrap, and live-cluster validation remain. | `tests/e2e/drivers/openshift.sh`, `tests/e2e/openshift_driver_test.sh` | OS-W1 | - -The other 12 OpenShift Development requirements remain missing and are intentionally outside this first slice: lifecycle driver/up-down, namespace isolation, Keycloak namespace deployment, component swaps, legacy-script consolidation, ephemeral CI, access handoff, blessed-overlay reconciliation, CI workflow, infrastructure prerequisites, cluster-scoped permissions, and security/RBAC parity. +| OS-1 | Cluster Lifecycle Driver Abstraction | Present | Kind targets keep current behavior via a thin wrapper around `scripts/kind/` | `scripts/cluster/`, `Makefile` | OS-W2 | +| OS-2 | OpenShift Lifecycle Up and Down | Present | `make openshift-up` / `openshift-down` / `openshift-status`. `openshift-teardown` is an alias of down (no cluster to destroy). Down deletes the platform project and `${name}-keycloak`. | `scripts/cluster/drivers/openshift.sh` | OS-W2 | +| OS-3 | Ephemeral Namespace Isolation | Present | Current `oc project -q` by default; `OPENSHIFT_NAMESPACE` overrides. Unlabeled existing projects are the developer's chosen target (no prompt; namespace labeling is best-effort). Refuse other HyperShell env ids and reserved names. Down tries `oc delete project` for the platform and `-keycloak` projects; if forbidden, deletes HyperShell resources (including unlabeled Keycloak) in both. Labels are not a delete gate. | `scripts/cluster/lib.sh`, `drivers/openshift.sh` | OS-W2 | +| OS-4 | Keycloak Namespace | Present | `${OPENSHIFT_NAMESPACE}-keycloak` via `oc new-project`; apply while that project is selected, then switch back; OIDC issuer from Keycloak Route; `KC_HOSTNAME` is host-only; `keycloak-allow-platform` NetworkPolicy lets platform pods reach JWKS/Admin API; console redirect URIs and API seeding use host `curl` against Routes (API server image has no curl) | `rewrite-namespaces.py`, `drivers/openshift.sh`, `deploy/openshift/keycloak-networkpolicy.yaml` | OS-W2 | +| OS-5 | Component Swap on OpenShift | Present | Build, push immutable commit+namespace identity, per-namespace `.openshift-swaps/`, reconcile preserves swaps | `drivers/openshift.sh`, `Makefile` | OS-W2 | +| OS-E2E-1 | OpenShift E2E Driver | Partial | Manual `make e2e` / `make e2e-performance` against a pre-deployed cluster. Out of scope for OS-W2. | `tests/e2e/drivers/openshift.sh` | OS-W1 | +| OS-7 | E2E Script Consolidation | Missing | Intentionally deferred: not local-dev lifecycle | `components/pr-test/` | Future | +| OS-8 | Ephemeral CI Environment Provisioning | Missing | Intentionally deferred: not local-dev lifecycle | - | Future | +| OS-9 | Environment Access Handoff | Missing | Intentionally deferred: CI-only | - | Future | +| OS-10 | Blessed OpenShift Overlay | Partial | Namespace parameterization, Routes, SCC RoleBindings; gateway base domain discovered from the shared Gateway listener (not `GATEWAY_API_BASE_DOMAIN`). Drift-check CI job deferred. | `deploy/openshift/`, `rewrite-namespaces.py` | OS-W2 | +| OS-11 | OpenShift CI Workflow Shape | Missing | Intentionally deferred: not local-dev lifecycle | - | Future | +| OS-12 | Cluster Infrastructure Prerequisites | Present | `make openshift-up` fails fast when the shared Gateway is missing or not Programmed. GatewayClass is cluster-scoped and not GET-checked (developers typically cannot read it). | `drivers/openshift.sh` `check_infrastructure` | OS-W2 | +| OS-13 | Cluster-Scoped Permissions + SCC/RBAC posture | Present | Namespace-scoped SCC *use* RoleBindings; per-namespace `bind` ClusterRoleBinding; overlay keeps `RBAC_ENFORCE=true`. E2E RBAC scenarios stay with the e2e suite. | `deploy/openshift/scc.yaml`, `drivers/openshift.sh` | OS-W2 | + +Local-dev lifecycle (`make openshift-up` / `down` / component swaps) is implemented. E2E driver completion beyond the OS-W1 manual slice, legacy `pr-test` consolidation, ephemeral CI, access handoff, overlay drift CI, and the OpenShift e2e workflow remain out of scope for this wave. ### openshell-gateway-service-accounts.spec.md @@ -593,6 +605,10 @@ Added `E2E_MODE=short|long` step tagging in `e2e-openshell.sh` (long remains the Implemented `tests/e2e/drivers/openshift.sh` for an already-deployed environment: namespace-scoped API and Keycloak Route discovery, Gateway API endpoint/readiness checks, gateway base-domain discovery from the running controller Deployment, `oc` selection, shared per-gateway Keycloak role flows, and TLS verification through the system trust store or an extracted private CA. The shared suite now routes direct HTTP calls through the driver TLS seam. Manual runs require only the current `oc` context and `OPENSHIFT_NAMESPACE`. No OpenShift cluster lifecycle, namespace creation, overlay reconciliation, bootstrap, or CI automation is included. +### Wave OS-W2: OpenShift local-dev lifecycle (up/down/swap) ✅ + +Implemented `scripts/cluster/` with a driver model. `make kind-*` wraps today's `scripts/kind/` with no behavior change. `make openshift-up` deploys `kustomize build deploy/openshift/` into an ephemeral namespace group (`OPENSHIFT_NAMESPACE` + `${OPENSHIFT_NAMESPACE}-keycloak`), stamps ownership labels, refuses foreign namespaces, fails fast when the shared Gateway is missing, reads the gateway base domain from that Gateway's listener hostname, seeds Fleet/ManagedCluster/GatewayRelease/ManagedDatabase/Gateway, and prints Routes. `make openshift-down` deletes only owned namespaces. Component swaps (`make openshift-api-server-up` and siblings) build, push an immutable commit+namespace identity to the internal registry, record per-namespace state in `.openshift-swaps/`, and are preserved across reconcile. Overlay: web-console + Keycloak Routes; SCC *use* is namespace-scoped RoleBindings. E2E, CI, `pr-test` consolidation, and overlay drift CI were explicitly out of scope. + ### Wave R1-R8: RBAC COMPLETED | Wave | Scope | Status | @@ -753,4 +769,7 @@ label-selected pod informer. | 2026-08-21 | working tree | Executed HYPERSHELL-49 SA-W4 | pending final recount | Extended the CLI generator for the nested gateway collection; added create/list/get/revoke/delete commands, explicit mode-0600 one-time credential output, expiration handling, workspace guidance, and secret-redaction tests. | | 2026-08-25 | working tree | Executed PERF-W1 (e2e-testing performance features) | 82% | Short/long `E2E_MODE` in the e2e suite; infra-agnostic performance harness with batched scale-up, canary checkpoints, functional gate, optional SLOs, schema_version=1 JSON history, and `make e2e-performance` / `make e2e-performance-report`. OpenShift driver still belongs to HYPERSHELL-44. | | 2026-08-26 | working tree | Executed OS-W1 (manual OpenShift e2e driver slice) | 78% | Added the OpenShift driver needed to run the existing e2e and performance harnesses against a pre-deployed cluster; lifecycle, overlay, bootstrap, and CI requirements remain missing. | +| 2026-08-27 | working tree | Executed OS-W2 (OpenShift local-dev up/down/swap) | 77% | Lifecycle driver model, `make openshift-up`/`down`/`status`, namespace isolation, Keycloak namespace, component swaps via internal registry, overlay namespace parameterization. E2E/CI left deferred. | +| 2026-08-31 | working tree | OpenShift Keycloak NetworkPolicy for JWKS | 77% | `keycloak-allow-platform` lets platform pods reach Keycloak TCP/8080 across the default-deny project policies so API server JWKS load and Admin API calls succeed. | +| 2026-08-31 | working tree | OpenShift console redirect URI + Route seeding | 77% | Host `curl` against Keycloak/API Routes registers the web-console `/auth/callback` (realm import only had Kind localhost URIs) and seeds the API. The API server image has no curl, so `oc exec curl` never obtained tokens. | | 2026-09-01 | feecbcb, da771fb | Reconciled commit-driven stale doc gaps | 79% (unchanged) | Two recent commits removed hardcoded image defaults (`GATEWAY_IMAGE`/`GATEWAY_SUPERVISOR_IMAGE` now required env vars, no fallback) and unified deploy paths (deleted `components/api-server/deploy/*`, using repo-root `deploy/` as single source of truth). Updated 7 docs: `skills/deploy/ibm-cluster/SKILL.md` (image refs, path, namespace, image-var explanation), `skills/deploy/gcp-cluster/SKILL.md` (path fix, RBAC ref), `skills/deploy/deploy-cluster/SKILL.md` (full rewrite: Keycloak bootstrap, `hypershell-api-config` Secret creation, CNPG database, OIDC/JWT security, troubleshooting for missing Secret), `skills/tooling/update-openshell/SKILL.md` (grep patterns for new image names, search path fixes), `skills/RECONCILE.md` (skill directory tree, this log entry), `README.md` (env var rows, namespace refs), `specs/platform/openshift-development.spec.md` (deploy/ directory layout, overlay limitations note). Overlay gaps surfaced: `deploy/openshift/` requires manually-created `hypershell-api-config` Secret (missing from repo; documented in deploy-cluster), hardcoded domain placeholder, missing Keycloak Route on OpenShift. Marked as known limitations in specs. | diff --git a/specs/platform/e2e-testing.spec.md b/specs/platform/e2e-testing.spec.md index 634afbb6..e8d5cc94 100644 --- a/specs/platform/e2e-testing.spec.md +++ b/specs/platform/e2e-testing.spec.md @@ -3,8 +3,8 @@ **Date:** 2026-08-10 **Status:** Draft **Jira:** HYPERSHELL-18 -**Related:** `local-development.spec.md` -- Kind cluster setup; - `openshift-development.spec.md` (HYPERSHELL-44) -- `make openshift-*` lifecycle, blessed `deploy/openshift/` overlay, cluster infrastructure bootstrap, OpenShift CI (this spec partially implements HYPERSHELL-44 by delivering the OpenShift e2e driver); +**Related:** `local-development.spec.md` -- Kind cluster setup and the shared `scripts/cluster/` lifecycle dispatcher; + `openshift-development.spec.md` (HYPERSHELL-44) -- `make openshift-*` lifecycle, blessed `deploy/openshift/` overlay, cluster infrastructure bootstrap, OpenShift CI (this spec owns the e2e driver interface and the OpenShift e2e driver; that spec owns bring-up); `control-plane.spec.md` -- reconciler behavior; `openshell-gateway-routing.spec.md` -- GRPCRoute provisioning; `openshell-gateway-namespace-gc.spec.md` -- gateway deletion + namespace GC; @@ -20,9 +20,9 @@ HyperShell also needs a **performance test**. The performance test measures how ### Scope -This spec covers the **e2e driver interface contract** (for all targets), the **Kind driver**, the **OpenShift e2e driver** (delivered here as a partial implementation of HYPERSHELL-44), the **Kind-based CI workflow**, and the **infra-agnostic performance test**. +This spec covers the **e2e driver interface contract** (for all targets), the **Kind driver**, the **OpenShift e2e driver**, the **Kind-based CI workflow**, and the **infra-agnostic performance test**. -This spec **partially implements** `openshift-development.spec.md` (HYPERSHELL-44): it delivers the driver interface contract for all targets and the **OpenShift e2e driver** (`tests/e2e/drivers/openshift.sh`) so a user can run `make e2e` and `make e2e-performance` **manually** against any OpenShift cluster the user is already logged in to (via `oc login`) -- the target environment for scale and performance testing. The remainder of HYPERSHELL-44 -- the `make openshift-*` lifecycle, the **blessed `deploy/openshift/` overlay**, the **cluster infrastructure bootstrap**, and the **automated OpenShift CI workflow** -- stays in that spec and is not duplicated here. Because the driver ships with this spec, the OpenShift-run requirements below are meetable here rather than deferred. +This spec owns the driver interface contract and the **OpenShift e2e driver** (`tests/e2e/drivers/openshift.sh`) so a user can run `make e2e` and `make e2e-performance` **manually** against any OpenShift cluster the user is already logged in to (via `oc login`) -- the target environment for scale and performance testing. Bring-up is a precondition: `make openshift-up` (specified in `openshift-development.spec.md`) deploys the blessed `deploy/openshift/` overlay into the current `oc` project (`OPENSHIFT_NAMESPACE` overrides), companion `${OPENSHIFT_NAMESPACE}-keycloak`, and the per-environment `${OPENSHIFT_NAMESPACE}-dev-*` cluster-scoped RBAC. This spec does not duplicate that lifecycle. An automated OpenShift CI job and the consolidation of `components/pr-test/e2e-openshell.sh` remain in HYPERSHELL-44. An automated OpenShift CI job is **out of scope** here; it belongs to HYPERSHELL-44. In this spec, OpenShift runs are manual and on-demand, and only the Kind e2e workflow runs in CI. @@ -36,7 +36,7 @@ tests/e2e/e2e-openshell.sh (infra-agnostic test logic) └── sources driver via E2E_INFRA_DRIVER (required) │ ├── tests/e2e/drivers/kind.sh (this spec) - └── tests/e2e/drivers/openshift.sh (this spec; partial HYPERSHELL-44) + └── tests/e2e/drivers/openshift.sh (this spec) ``` The driver model separates test logic from infrastructure mechanics. The main test script calls a fixed set of driver functions; each driver implements those functions for its target infrastructure. Adding a new infrastructure target requires only a new driver file. @@ -49,7 +49,7 @@ Each driver exports shell functions that abstract infrastructure-specific operat |----------|---------|---------------------|--------------------------| | `discover_api_host` | Find the HyperShell API server URL | HTTPRoute hostname `api.hypershell.localhost` or port-forward to `svc/hypershell-api-server` | `oc get route hypershell-api -o jsonpath='{.spec.host}'` | | `discover_gateway_endpoint` | Find the gateway gRPC endpoint | GRPCRoute hostname `.gw.localhost` via Gateway status address | GRPCRoute hostname via shared Gateway `Programmed=True` (Gateway API, not a per-gateway Route) | -| `get_cluster_domain` | Get the base domain for constructing gateway DNS names | `gw.localhost` (static, matching `GATEWAY_API_BASE_DOMAIN` in `deploy/kind/`) | Configured `GATEWAY_API_BASE_DOMAIN` (not the cluster apps domain) | +| `get_cluster_domain` | Get the base domain for constructing gateway DNS names | `gw.localhost` (static, matching `GATEWAY_API_BASE_DOMAIN` in `deploy/kind/`) | Gateway base domain derived from the shared Gateway listener hostname -- the same value `make openshift-up` sets on the control plane. Not a developer-supplied `GATEWAY_API_BASE_DOMAIN`, and not the cluster apps domain | | `get_cli_binary` | Return the Kubernetes CLI binary path | `kubectl` | `oc` | | `wait_for_gateway_route` | Block until the gateway is externally reachable | Check Gateway API Gateway status conditions and GRPCRoute parent status | Check Gateway `Programmed=True` and GRPCRoute parent `Accepted=True` | | `acquire_oidc_token` | Obtain an OIDC access token for a given user, stored in `_OIDC_ACCESS_TOKEN` for `api_curl` to use | Resource-owner password grant against Keycloak at `keycloak.hypershell.localhost`, trusting the Kind self-signed CA (`curl -k`) | Resource-owner password grant against the HyperShell Keycloak at its Route in the `${OPENSHIFT_NAMESPACE}-keycloak` namespace, in the `hypershell` realm, trusting the cluster CA | @@ -119,7 +119,9 @@ deploy/ kustomization.yaml openshift/ ← OpenShift overlay (extends base) kustomization.yaml ← references ../base - route.yaml + route.yaml ← API + web-console Routes + keycloak-route.yaml + keycloak-networkpolicy.yaml ← platform pods may reach Keycloak JWKS/Admin API scc.yaml certificates.yaml networkpolicies.yaml @@ -170,7 +172,7 @@ The e2e test framework SHALL isolate infrastructure-specific logic into driver s ### Requirement: Driver Interface Contract -Each driver script SHALL export the following shell functions. The main test script SHALL call only these functions for infrastructure-specific operations. A driver that does not implement all required functions SHALL cause the test script to exit with an error at startup. This spec defines the contract for all drivers and covers the Kind driver implementation. The OpenShift implementation of this contract (the `oc` commands, Route discovery, cluster-domain lookup, and OIDC issuer derivation) is specified in `openshift-development.spec.md` (HYPERSHELL-44); the table below is the contract it implements. +Each driver script SHALL export the following shell functions. The main test script SHALL call only these functions for infrastructure-specific operations. A driver that does not implement all required functions SHALL cause the test script to exit with an error at startup. This spec defines the contract for all drivers and covers the Kind driver implementation. The OpenShift implementation of this contract (the `oc` commands, Route discovery, gateway-base-domain lookup from the shared Gateway listener, and OIDC issuer derivation from the Keycloak Route) is specified in `openshift-development.spec.md` (HYPERSHELL-44); the table below is the contract it implements. #### Scenario: API Host Discovery -- Kind @@ -207,15 +209,15 @@ Each driver script SHALL export the following shell functions. The main test scr - AND verify the corresponding GRPCRoute's parent status reports `Accepted=True` - AND return success when both conditions are met or fail after `E2E_PROVISION_TIMEOUT` seconds -The OpenShift driver implements the same ten functions with OpenShift constructs (Route host for `discover_api_host`, GRPCRoute hostname via the shared Gateway with `Programmed=True` for `discover_gateway_endpoint`, the configured `GATEWAY_API_BASE_DOMAIN` for `get_cluster_domain`, `oc` for `get_cli_binary`, Gateway `Programmed=True` plus GRPCRoute parent `Accepted=True` for `wait_for_gateway_route`, the HyperShell Keycloak reached at its Route in the `${OPENSHIFT_NAMESPACE}-keycloak` namespace for `acquire_oidc_token` and `api_curl`, and that same Keycloak's admin API for the `assign_gateway_client_role`, `assign_realm_role`, and `acquire_gateway_token_with_role` role helpers), as the interface table above shows. This spec delivers the OpenShift driver as a partial implementation of `openshift-development.spec.md` (HYPERSHELL-44), which is what makes the manual OpenShift runs defined next implementable here; HYPERSHELL-44 owns the surrounding `make openshift-*` lifecycle, `deploy/openshift/` overlay, cluster bootstrap, and automated CI. +The OpenShift driver implements the same ten functions with OpenShift constructs (Route host for `discover_api_host`, GRPCRoute hostname via the shared Gateway with `Programmed=True` for `discover_gateway_endpoint`, the gateway base domain `make openshift-up` derived from the shared Gateway listener hostname for `get_cluster_domain`, `oc` for `get_cli_binary`, Gateway `Programmed=True` plus GRPCRoute parent `Accepted=True` for `wait_for_gateway_route`, the HyperShell Keycloak reached at its Route in the `${OPENSHIFT_NAMESPACE}-keycloak` namespace for `acquire_oidc_token` and `api_curl`, and that same Keycloak's admin API for the `assign_gateway_client_role`, `assign_realm_role`, and `acquire_gateway_token_with_role` role helpers), as the interface table above shows. `openshift-development.spec.md` owns bring-up (`make openshift-up`, the overlay, cluster bootstrap); this spec owns the driver the suite calls after that environment exists. Automated OpenShift CI remains in HYPERSHELL-44. ### Requirement: Custom OpenShift Runs -Each target SHALL default `E2E_INFRA_DRIVER` to `kind` and SHALL honor a command-line override. A user SHALL be able to run `make e2e` and `make e2e-performance` **manually** against any OpenShift cluster, so scale and performance testing can target a real OpenShift environment: a user SHALL run `E2E_INFRA_DRIVER=openshift make e2e` or `E2E_INFRA_DRIVER=openshift make e2e-performance` against the cluster their current `oc` context selects. This spec delivers the OpenShift driver (`tests/e2e/drivers/openshift.sh`) as a partial implementation of HYPERSHELL-44, so these runs are implementable here rather than deferred to that spec. These OpenShift runs SHALL NOT create a cluster and SHALL NOT create a namespace beyond the gateways the suite provisions; the environment is a precondition. +Each target SHALL default `E2E_INFRA_DRIVER` to `kind` and SHALL honor a command-line override. A user SHALL be able to run `make e2e` and `make e2e-performance` **manually** against any OpenShift cluster, so scale and performance testing can target a real OpenShift environment: a user SHALL run `E2E_INFRA_DRIVER=openshift make e2e` or `E2E_INFRA_DRIVER=openshift make e2e-performance` against the cluster their current `oc` context selects. These OpenShift runs SHALL NOT create a cluster and SHALL NOT create a namespace beyond the gateways the suite provisions; the environment is a precondition. -**Preconditions (owned by HYPERSHELL-44).** These runs assume HyperShell is already deployed on the cluster through `kustomize build deploy/openshift/` (for example via `make openshift-up`), and that the cluster infrastructure bootstrap (shared Gateway, GatewayClass, certificate issuer, wildcard certificate) is in place per `openshift-development.spec.md`. The suite SHALL fail with a clear error, not a broken run, when the API Route or the gateway infrastructure is absent. +**Preconditions (owned by `openshift-development.spec.md`).** These runs assume HyperShell is already deployed on the cluster through `make openshift-up` (`kustomize build deploy/openshift/` mapped into the current `oc` project, or `OPENSHIFT_NAMESPACE`). That bring-up creates the companion `${OPENSHIFT_NAMESPACE}-keycloak` project, applies Routes for the API, web console, and Keycloak, applies `keycloak-allow-platform` so platform pods can reach JWKS, applies per-environment ClusterRoles and ClusterRoleBindings named `${OPENSHIFT_NAMESPACE}-dev-*`, and applies the privileged SCC RoleBinding `hypershell-sandbox-scc`. The cluster infrastructure bootstrap (shared Gateway, GatewayClass, certificate issuer, wildcard certificate) is in place per `openshift-development.spec.md`. The suite SHALL fail with a clear error, not a broken run, when the API Route or the gateway infrastructure is absent. -**Driver behavior needed for parity.** For the shared suite to pass on OpenShift, the OpenShift driver SHALL derive the OIDC issuer and related OIDC variables from the running cluster's domain (not the Kind default `keycloak.hypershell.localhost`), and SHALL provide the same Keycloak admin and role-assignment helpers the Kind driver provides, so the RBAC areas (developer and platform-admin) run unchanged. The OpenShift deployment SHALL enforce RBAC (`RBAC_ENFORCE=true`) and SHALL keep the OpenShift SCC posture (per-namespace privileged SCC for sandbox pods), so the sandbox and RBAC areas behave the same as on Kind. These behaviors are specified in HYPERSHELL-44; this spec only depends on them. +**Driver behavior needed for parity.** For the shared suite to pass on OpenShift, the OpenShift driver SHALL derive the OIDC issuer from the Keycloak Route in `${OPENSHIFT_NAMESPACE}-keycloak` (not the Kind default `keycloak.hypershell.localhost`), SHALL return `get_cluster_domain` from the same shared-Gateway listener hostname `make openshift-up` used, and SHALL provide the same Keycloak admin and role-assignment helpers the Kind driver provides, so the RBAC areas (developer and platform-admin) run unchanged. The OpenShift deployment SHALL enforce RBAC (`RBAC_ENFORCE=true`) and SHALL keep the OpenShift SCC posture (per-namespace privileged SCC for sandbox pods), so the sandbox and RBAC areas behave the same as on Kind. These behaviors are specified in `openshift-development.spec.md`; this spec only depends on them. **Namespace GC timing.** Area 11 exercises the periodic namespace reaper. To make it pass on OpenShift without waiting the production GC defaults (5m sweep / 10m grace), the OpenShift deployment SHOULD set shortened `GATEWAY_NAMESPACE_GC_INTERVAL` and `GATEWAY_NAMESPACE_GC_GRACE_PERIOD` (as the Kind overlay does), or the user SHOULD raise `E2E_ORPHAN_GC_TIMEOUT` and `E2E_GC_TIMEOUT` to fit the cluster's configured timing. @@ -223,8 +225,8 @@ Each target SHALL default `E2E_INFRA_DRIVER` to `kind` and SHALL honor a command #### Scenario: e2e Against OpenShift -- GIVEN the OpenShift driver delivered by this spec is present at `tests/e2e/drivers/openshift.sh` -- AND a user is logged in to an OpenShift cluster with HyperShell deployed (`deploy/openshift/`) +- GIVEN the OpenShift driver is present at `tests/e2e/drivers/openshift.sh` +- AND a user is logged in to an OpenShift cluster with HyperShell deployed via `make openshift-up` - AND the cluster infrastructure bootstrap is in place per `openshift-development.spec.md` - WHEN the user runs `E2E_INFRA_DRIVER=openshift make e2e` - THEN the suite SHALL run against that cluster using the OpenShift driver @@ -232,8 +234,8 @@ Each target SHALL default `E2E_INFRA_DRIVER` to `kind` and SHALL honor a command #### Scenario: Performance Against OpenShift -- GIVEN the OpenShift driver delivered by this spec is present at `tests/e2e/drivers/openshift.sh` -- AND a user is logged in to an OpenShift cluster with HyperShell deployed +- GIVEN the OpenShift driver is present at `tests/e2e/drivers/openshift.sh` +- AND a user is logged in to an OpenShift cluster with HyperShell deployed via `make openshift-up` - WHEN the user runs `E2E_INFRA_DRIVER=openshift make e2e-performance` - THEN the performance harness SHALL run against that cluster using the OpenShift driver - AND it SHALL provision the perf fleet on that cluster and report metrics @@ -615,7 +617,8 @@ The `deploy/` directory SHALL use a kustomize base/overlay structure to support - GIVEN `deploy/openshift/kustomization.yaml` references `../base` as a resource - WHEN `kustomize build deploy/openshift/` is executed - THEN the output SHALL include all base resources -- AND OpenShift-specific resources: Route for the API server with edge TLS termination, SecurityContextConstraints bindings +- AND OpenShift-specific resources: Routes for the API server, web console, and Keycloak with edge TLS termination; SecurityContextConstraints RoleBindings; `keycloak-allow-platform` NetworkPolicy so platform pods can reach Keycloak on TCP/8080 +- AND `make openshift-up` SHALL rewrite overlay namespaces so `hypershell-system` maps to `OPENSHIFT_NAMESPACE` and `keycloak` maps to `${OPENSHIFT_NAMESPACE}-keycloak`, prefix cluster-scoped RBAC names with `${OPENSHIFT_NAMESPACE}-dev-`, and set the control plane `GATEWAY_API_BASE_DOMAIN` from the shared Gateway listener hostname #### Scenario: Base Resource Propagation @@ -685,7 +688,9 @@ deploy/ kustomization.yaml openshift/ -- OpenShift overlay (extends base) kustomization.yaml -- references ../base - route.yaml + route.yaml -- API + web-console Routes + keycloak-route.yaml + keycloak-networkpolicy.yaml scc.yaml certificates.yaml networkpolicies.yaml @@ -702,7 +707,8 @@ deploy/ | Env Var | Default | Description | |---------|---------|-------------| -| `E2E_INFRA_DRIVER` | (required) | Infra driver to use: `kind`, `openshift` (OpenShift driver per HYPERSHELL-44) | +| `E2E_INFRA_DRIVER` | (required) | Infra driver to use: `kind` or `openshift` | +| `OPENSHIFT_NAMESPACE` | current `oc project` | Platform namespace the OpenShift driver and `make openshift-up` target; Keycloak is `${OPENSHIFT_NAMESPACE}-keycloak` | | `E2E_NAMESPACE` | `openshell-e2e` | Namespace for e2e test resources (gateway deployment) | | `E2E_GATEWAY_NAME` | `e2e-gw` | Gateway name for the e2e test | | `E2E_MODE` | `long` | Run depth: `long` runs every step, `short` runs the essential steps of each area (see [E2E Short and Long Modes](#requirement-e2e-short-and-long-modes)) | @@ -724,7 +730,7 @@ deploy/ ### Requirement: OIDC Authentication in E2E Tests -The e2e test suite SHALL run with OIDC authentication enabled. The CI workflow SHALL deploy the Kind cluster with `KIND_ENABLE_OIDC=true`. All API calls SHALL be authenticated with a Bearer token obtained from Keycloak. This ensures e2e tests exercise the same authentication path as production. +The e2e test suite SHALL run with OIDC authentication enabled. `make kind-up` enables OIDC unconditionally (`API_ENV=development_oidc`); there is no `KIND_ENABLE_OIDC` toggle. All API calls SHALL be authenticated with a Bearer token obtained from Keycloak. This ensures e2e tests exercise the same authentication path as production. The test suite SHALL verify OIDC integration as part of its standard flow: 1. Acquire a token from Keycloak and authenticate all API calls @@ -764,7 +770,7 @@ The test suite SHALL verify OIDC integration as part of its standard flow: - GIVEN the CI e2e workflow - WHEN the Kind cluster is created -- THEN `make kind-up` SHALL be invoked with `KIND_ENABLE_OIDC=true` +- THEN `make kind-up` SHALL enable OIDC (the Kind overlay always sets `API_ENV=development_oidc`) ### Requirement: Web Console Distributed Trace Verification @@ -835,7 +841,7 @@ The system SHALL provide a `make e2e-performance` target. The target SHALL run ` #### Scenario: OpenShift Run - GIVEN a user is logged in to an OpenShift cluster with HyperShell deployed -- AND the `openshift` driver is present at `tests/e2e/drivers/openshift.sh` (delivered by this spec) +- AND the `openshift` driver is present at `tests/e2e/drivers/openshift.sh` - WHEN the user runs `E2E_INFRA_DRIVER=openshift make e2e-performance` - THEN the harness SHALL run against the OpenShift cluster with no change to the harness code - AND all infrastructure operations SHALL use the OpenShift driver (`oc`, Routes) @@ -846,7 +852,7 @@ The performance harness (`tests/e2e/e2e-performance.sh`) SHALL be infrastructure The harness SHALL obtain the seeded cluster, release, and managed database ids the same way the e2e suite does: it SHALL query the API through `api_curl` and reuse the shared seeding helpers in `tests/e2e/lib.sh`, never hardcoding ids. When `E2E_SEED_CLUSTER_NAME` / `E2E_SEED_RELEASE_NAME` are set, discovery SHALL select the matching name; when they are unset it SHALL take the first list item (the single-seed Kind/CI layout). On `E2E_INFRA_DRIVER=kind` those names SHALL default to the `make kind-up` seeds (`local-kind`, `dev-release`). Every diagnostic or resource-inspection command SHALL invoke the Kubernetes CLI through `$(get_cli_binary)`, so it resolves to `kubectl` on Kind and `oc` on OpenShift with no change to the harness. -The OpenShift driver is delivered by this spec as a partial implementation of `openshift-development.spec.md` (HYPERSHELL-44); the performance harness uses it for OpenShift runs (see [Scope](#scope)). The harness SHALL contain no infra-specific code: it works with either driver with no change. OpenShift runs are manual and on-demand; the performance test is not wired into CI for any target (see [Design Decisions](#design-decisions)). +The OpenShift driver is specified alongside this contract in `openshift-development.spec.md`; the performance harness uses it for OpenShift runs (see [Scope](#scope)). The harness SHALL contain no infra-specific code: it works with either driver with no change. OpenShift runs are manual and on-demand; the performance test is not wired into CI for any target (see [Design Decisions](#design-decisions)). #### Scenario: Driver Not Set @@ -1285,7 +1291,7 @@ On failure, the harness SHALL collect diagnostics that explain resource pressure | `E2E_PERF_CSV` | `0` | Set to `1` to also append each run to `/history.csv` | | `E2E_PERF_MIN_SUCCESS_RATE` | (unset) | Optional SLO: min provisioning success rate percent; below this fails the run | | `E2E_PERF_MAX_PROVISION_P99` | (unset) | Optional SLO: max p99 time-to-`Running` seconds; above this fails the run | -| `E2E_INFRA_DRIVER` | (required; `make e2e-performance` defaults to `kind`) | Infra driver: `kind`, `openshift` (OpenShift driver per HYPERSHELL-44) | +| `E2E_INFRA_DRIVER` | (required; `make e2e-performance` defaults to `kind`) | Infra driver: `kind` or `openshift` | | `E2E_SKIP_CLEANUP` | `0` | Set to `1` to keep the perf fleet after the run | **Capacity note:** a small Kind cluster cannot run hundreds of gateways. Each gateway provisions a deployment, a service, a TLS secret, a certgen job, a per-gateway Keycloak client, and a managed namespace. A run also stands up the canary and the functional gateway, so the cluster carries `E2E_PERF_GATEWAY_COUNT + 2` gateway stacks at peak: the default of 5 means 7 stacks, which fits a typical Kind cluster. Keep the total modest on Kind (roughly `count + 2` at or below 10). Use a larger count on an OpenShift cluster that has spare capacity. The harness reports resource pressure on failure so a user can find the ceiling. @@ -1304,7 +1310,7 @@ On failure, the harness SHALL collect diagnostics that explain resource pressure | e2e workflow skips for irrelevant changes | SDK-only or docs-only PRs do not affect the e2e path. Skipping avoids CI time and Konflux build overhead. The `detect-components.sh` infrastructure tracks `api_server`, `control_plane`, `pr_test`, and `e2e` component paths for "should we re-run e2e" decisions. Separately, Konflux image builds only trigger on changes under `components//` source paths -- the workflow checks the actual diff to distinguish e2e-relevant infrastructure changes (which use baseline images) from source changes (which require Konflux-built images) | | `make kind-up` accepts image overrides | Passing `IMAGE_TAG=` or per-component image variables to `make kind-up` allows CI to deploy Konflux-built images directly without a separate load step. Developers can also use this to test specific image versions locally | | Backward-compatible migration | The refactoring does not change `make kind-up`. `scripts/kind/up.sh` can be migrated to use `kustomize build deploy/kind/` incrementally. The spec defines the target state; the migration path is incremental | -| HYPERSHELL-18 partially implements HYPERSHELL-44 | This spec delivers two slices of `openshift-development.spec.md` (HYPERSHELL-44): the driver interface contract (both columns) and the OpenShift e2e driver (`tests/e2e/drivers/openshift.sh`), so `make e2e` and `make e2e-performance` run manually against any OpenShift cluster for scale and performance testing. Delivering the driver here keeps the manual-run requirement meetable instead of gating it on the rest of HYPERSHELL-44. The remainder -- the `make openshift-*` lifecycle, the `deploy/openshift/` overlay, the cluster infrastructure bootstrap, and the automated OpenShift CI job -- stays in HYPERSHELL-44 and is not duplicated here | +| OpenShift e2e runs use `make openshift-up` as the environment | This spec owns the driver the suite calls. `openshift-development.spec.md` owns bring-up: `make openshift-up`, the `deploy/openshift/` overlay (Routes, Keycloak NetworkPolicy, SCC), namespace rewrite, `${OPENSHIFT_NAMESPACE}-dev-*` cluster RBAC, and cluster bootstrap. Automated OpenShift CI and `components/pr-test/` consolidation stay in HYPERSHELL-44 and are not duplicated here | | Env vars renamed with `E2E_` prefix | The existing `e2e-openshell.sh` uses `SANDBOX_TIMEOUT`, `PROVISION_TIMEOUT`, `SKIP_CLEANUP`, and `GATEWAY_NAMESPACE`. These are renamed to `E2E_SANDBOX_TIMEOUT`, `E2E_PROVISION_TIMEOUT`, `E2E_SKIP_CLEANUP`, and `E2E_NAMESPACE` to avoid namespace collisions with non-e2e configuration and make the e2e origin of these variables explicit | | CI uses `make kind-up`, not raw `kind create cluster` | Reuses the same cluster setup path developers use locally. Ensures the CI environment is identical to local development. Avoids a second "create a Kind cluster" implementation that could drift | | Performance harness reuses the e2e driver interface | The performance test needs the same cross-infrastructure portability as the e2e suite: run on Kind locally, run on any OpenShift cluster for on-demand load tests. Reusing the driver interface means the harness holds no infra-specific code and a new target needs only a new driver file. It also keeps one abstraction to maintain, not two | diff --git a/specs/platform/local-development.spec.md b/specs/platform/local-development.spec.md index 96da4bbf..4ad4edcc 100644 --- a/specs/platform/local-development.spec.md +++ b/specs/platform/local-development.spec.md @@ -11,7 +11,14 @@ HyperShell provides a single-command local development environment using Kind (K Developers selectively swap individual components with local builds using per-component targets. The baseline cluster runs pre-built images pulled from the container registry; individual components are "swapped in" from local source as needed. Selective swapping converges to the current working tree state. -The same lifecycle model extends to OpenShift. The make target name selects the infrastructure: `make kind-up` deploys to Kind (this spec) and `make openshift-up` deploys the same components into an ephemeral OpenShift namespace. The per-component swap targets follow the same pattern. `openshift-development.spec.md` specifies the OpenShift lifecycle commands, the ephemeral-namespace model, and the OpenShift e2e driver. +The same lifecycle model extends to OpenShift. The make target name selects the +infrastructure: `make kind-up` deploys to Kind (this spec) and `make openshift-up` +deploys the same components into an ephemeral OpenShift namespace group. Kind +targets dispatch through `scripts/cluster/` with `CLUSTER_DRIVER=kind`, wrapping +`scripts/kind/` so Kind behavior does not change. The per-component swap targets +follow the same pattern. `openshift-development.spec.md` specifies the OpenShift +lifecycle commands, the ephemeral-namespace model, cluster-scoped RBAC, and the +OpenShift e2e driver. ## Components Deployed @@ -579,13 +586,14 @@ The system SHALL track which components have been swapped to local builds using The repository SHALL include a `DEVELOPMENT.md` guide that documents the local development environment. The guide SHALL cover: -- Prerequisites (Docker or Podman, Kind, kubectl) +- Prerequisites (Docker or Podman, Kind, kubectl; `oc` for the OpenShift path) - `make kind-up` quickstart with expected output - Per-component swap workflow (`make kind--up` / `make kind--down`) - Hot reload setup for the web console (`KIND_HOT_RELOAD=true`) - Environment variable reference (all `KIND_*`, `IMAGE_*`, and `CONTAINER_ENGINE` variables) - Keycloak configuration and `KIND_KEYCLOAK_URL` for external OIDC - Troubleshooting common issues (port conflicts, container engine not running, image pull failures) +- The OpenShift counterpart (`make openshift-up`, `make openshift--up`) in the same guide; that workflow is specified by `openshift-development.spec.md` The documentation SHALL be kept in sync with this spec. When a new Make target, environment variable, or component is added, the guide SHALL be updated in the same PR. @@ -593,9 +601,10 @@ The documentation SHALL be kept in sync with this spec. When a new Make target, - GIVEN a developer clones the repository - WHEN they look for local development instructions - THEN `DEVELOPMENT.md` SHALL exist and describe how to set up and use the Kind environment +- AND it SHALL document `make openshift-up` against an existing OpenShift cluster #### Scenario: Documentation Stays Current -- GIVEN a PR adds or changes a `kind-*` Make target or environment variable +- GIVEN a PR adds or changes a `kind-*` or `openshift-*` Make target or environment variable - WHEN the PR is reviewed - THEN the reviewer SHALL verify that `DEVELOPMENT.md` is updated to reflect the change @@ -796,7 +805,7 @@ The system SHALL deploy a Jaeger all-in-one instance in the local environment an ## Make Targets Summary -All targets operate on `KIND_NAMESPACE` (default: `hypershell-system`). +All Kind targets operate on `KIND_NAMESPACE` (default: `hypershell-system`). OpenShift uses the same target names with an `openshift-` prefix (`make openshift-up`, `make openshift-api-server-up`, and siblings); those commands are specified in `openshift-development.spec.md`. `make kind-teardown` destroys the Kind cluster. `make openshift-teardown` is the same as `make openshift-down` -- there is no OpenShift cluster to destroy. | Target | Behavior | |--------|----------| @@ -848,4 +857,4 @@ All targets operate on `KIND_NAMESPACE` (default: `hypershell-system`). | OIDC only, no mTLS | Team agreed to drop mTLS client auth; OIDC is the recommended auth mode for Kubernetes deployments per upstream docs | | TLS always enabled | BackendTLSPolicy re-encrypts traffic from the networking Gateway to the pod (see Gateway API Routing section); the gateway must serve TLS even in local environments. cert-manager issues a self-signed CA for both the wildcard listener cert and the pod's server cert | | Configurable `IMAGE_REGISTRY` and `IMAGE_TAG` | Allows teams to test against different builds or staging registries | -| Single root Makefile | All targets live in the root Makefile - build, test, codegen, and cluster lifecycle. Component-level Makefiles (`components/api-server/Makefile`, etc.) are deprecated; a single entrypoint eliminates indirection and makes `make ` discoverable. Kind cluster lifecycle shell logic lives in `scripts/kind/` (`lib.sh`, `up.sh`, `down.sh`, `teardown.sh`, `status.sh`, `build-images.sh`, `swap-component.sh`); the Makefile exports configuration and dispatches to these scripts. Output uses colored headers (`NO_COLOR` respected) | +| Single root Makefile | All targets live in the root Makefile - build, test, codegen, and cluster lifecycle. Component-level Makefiles (`components/api-server/Makefile`, etc.) are deprecated; a single entrypoint eliminates indirection and makes `make ` discoverable. Kind and OpenShift lifecycle dispatch through `scripts/cluster/` (`CLUSTER_DRIVER=kind` or `openshift`). The Kind driver wraps `scripts/kind/` (`lib.sh`, `up.sh`, `down.sh`, `teardown.sh`, `status.sh`, `build-images.sh`, `swap-component.sh`) without behavior change. The Makefile exports configuration and selects the driver by target name. Output uses colored headers (`NO_COLOR` respected) | diff --git a/specs/platform/openshift-development.spec.md b/specs/platform/openshift-development.spec.md index 67ad165d..a308e622 100644 --- a/specs/platform/openshift-development.spec.md +++ b/specs/platform/openshift-development.spec.md @@ -150,12 +150,13 @@ infrastructure. operations - AND the developer does not change the infrastructure-agnostic lifecycle logic -#### Scenario: OpenShift teardown removes the namespace group +#### Scenario: OpenShift teardown is an alias of down - GIVEN an OpenShift environment exists -- WHEN the framework calls `cluster_teardown` for the OpenShift target -- THEN the driver removes the environment namespace group -- AND the driver does not attempt to destroy the OpenShift cluster +- WHEN the developer runs `make openshift-teardown` or `make openshift-down` +- THEN both commands remove the environment namespace group (the platform + project and the `${OPENSHIFT_NAMESPACE}-keycloak` project) +- AND neither command attempts to destroy the OpenShift cluster ### Requirement: OpenShift Lifecycle Up and Down @@ -192,8 +193,19 @@ OIDC values for the environment, so that one command produces a working gateway the OpenShift workflow matches the Kind workflow. The `make openshift-down` command SHALL delete the applied manifests and SHALL -remove every namespace in the environment namespace group, subject to the ownership -check that the Ephemeral Namespace Isolation requirement defines. The +remove every project in the environment namespace group: the platform project +the developer selected with `OPENSHIFT_NAMESPACE` or the current `oc project`, +and the companion `${OPENSHIFT_NAMESPACE}-keycloak` project. `make +openshift-teardown` SHALL be the same command. OpenShift does not create the +cluster, so teardown cannot destroy it; the Kind-shaped target exists for +compatibility. Ownership labels are not a delete gate: typical developers +cannot patch Namespace objects, so down SHALL NOT require those labels. The +command SHALL still refuse reserved names and namespaces labeled as a different +HyperShell environment. The command SHALL wait until each project is gone before it reports success, +rather than return after it has only requested deletion. When `oc delete project` +is forbidden, the command SHALL delete HyperShell resources inside both projects +— including the bundled Keycloak workload, which is unlabeled — wait for those +deletes, and leave the projects. The `make openshift-status` command SHALL report the cluster, the environment namespaces, the pods, the services, the Routes, the Gateway status, and the component swap state, the same categories that `make kind-status` reports. @@ -213,12 +225,33 @@ prefix with `openshift`. - AND the scripts report the API Route, the web-console Route, and the Keycloak Route when the deployment is ready +#### Scenario: Console login and API seeding use the OpenShift Routes + +The API server image does not include `curl`. Token grants, Keycloak Admin +updates, and seed POSTs SHALL run from the developer machine against the +Keycloak and API Routes, the same way `make kind-up` talks to Keycloak through +its hostname rather than `oc exec`. The imported realm only allows +`https://console.hypershell.localhost` redirect URIs; the driver SHALL set +`hypershell-frontend` redirect URIs to the web-console Route origin +(`https:///auth/callback` and `https://`) +so the BFF authorization-code callback succeeds. Wildcard redirect URIs SHALL +NOT be registered. + +- GIVEN a developer runs `make openshift-up` +- WHEN the deployment is ready +- THEN `hypershell-frontend` redirect URIs include the web-console Route `/auth/callback` +- AND the driver obtained the seed API token from the Keycloak Route +- AND Keycloak accepts the BFF `redirect_uri` for that console host + #### Scenario: Remove the deployment - GIVEN a HyperShell deployment exists from `make openshift-up` -- WHEN the developer runs `make openshift-down` -- THEN the scripts remove the HyperShell resources from every namespace in the - environment +- WHEN the developer runs `make openshift-down` or `make openshift-teardown` +- THEN the scripts delete the platform project and the companion `-keycloak` project +- AND the command does not return until both projects are gone, or until project + deletion is forbidden and HyperShell resources in both projects have been removed +- AND when project deletion is forbidden, the scripts remove HyperShell + resources from both projects, including Keycloak - AND the scripts do not delete resources that belong to other environments or to cluster infrastructure @@ -250,25 +283,48 @@ so that the derived `${OPENSHIFT_NAMESPACE}-keycloak` namespace stays within the 63-character DNS-label limit for Kubernetes namespaces. The command SHALL validate the name and the derived name before it creates any resource, and SHALL stop with a clear error when either name is invalid. The scripts SHALL derive the `-keycloak` -namespace from that name (see the Keycloak Namespace requirement). The scripts SHALL -create the namespaces if they do not exist. - -The scripts SHALL stamp every namespace in the group, at creation, with an -ownership label that marks the namespace as HyperShell-owned and with an immutable -environment identifier that ties the namespace to one deployment, so that -`make openshift-status` and cleanup tooling can find every HyperShell namespace and -can tell which namespaces belong to the same environment. Before it deploys, the -command SHALL refuse to adopt an existing namespace whose ownership label or -environment identifier does not match the current environment, so that a deployment -cannot take over a namespace that another environment or another team owns. The -scripts SHALL derive per-tenant gateway hostnames from the gateway base domain (the -configured `GATEWAY_API_BASE_DOMAIN`) and the platform namespace, so that two -deployments on one cluster do not share a hostname. - -Before it deletes, `make openshift-down` SHALL verify the ownership label and the -environment identifier on each namespace and SHALL delete only namespaces that match -the current environment, so that it cannot delete unrelated workloads. The command -SHALL refuse a mismatch and report it. +namespace from that name (see the Keycloak Namespace requirement). Missing projects +SHALL be created with `oc new-project` (an OpenShift ProjectRequest), not +`oc create namespace`. `oc new-project` selects the new project as the current oc +project. After Keycloak is applied in the `-keycloak` project, the scripts SHALL +switch the current project back to the platform project for the rest of the +deployment. + +The developer selects the target with `OPENSHIFT_NAMESPACE` or the current +`oc project`. That selection is the authority for `make openshift-up`. The +command SHALL deploy into that namespace group without prompting, and SHALL +NOT require permission to patch namespace objects. Typical developer accounts +on a shared cluster can create resources inside a project and cannot label the +Namespace itself. + +When the account can patch namespaces, the scripts SHALL stamp every namespace +in the group with an ownership label that marks the namespace as HyperShell-owned +and with an immutable environment identifier that ties the namespace to one +deployment, so that `make openshift-status` and cleanup tooling can find every +HyperShell namespace and can tell which namespaces belong to the same +environment. When labeling is forbidden, the scripts SHALL warn and continue, +and SHALL recover a previously applied environment identifier from workload +labels when those labels are present. + +Before it deploys, the command SHALL refuse an existing namespace whose +ownership label and environment identifier mark it as a different HyperShell +environment, so that a deployment cannot take over a namespace that another +environment owns. An unlabeled existing project is not foreign: the developer +already pointed `make openshift-up` at it. Reserved names (`default`, `kube-*`, +`openshift-*`) SHALL still be refused. The scripts SHALL derive per-tenant +gateway hostnames from the gateway base domain (the configured +`GATEWAY_API_BASE_DOMAIN`) and the platform namespace, so that two deployments +on one cluster do not share a hostname. + +Before it deletes, `make openshift-down` SHALL use the same project selection as +`make openshift-up`. It SHALL refuse reserved names and namespaces whose +ownership labels mark them as a different HyperShell environment. An unlabeled +project is the developer's chosen target: the command SHALL attempt +`oc delete project` for the platform namespace and for +`${OPENSHIFT_NAMESPACE}-keycloak`. When project deletion is forbidden, the +command SHALL remove HyperShell resources inside both projects and SHALL leave +the projects. The command SHALL NOT require namespace labels in order to delete. +`make openshift-teardown` SHALL perform the same steps. #### Scenario: Two developers share one cluster @@ -283,8 +339,7 @@ SHALL refuse a mismatch and report it. - GIVEN two HyperShell environment namespace groups exist on one cluster - WHEN a developer runs `make openshift-down` for one environment -- THEN the scripts verify the ownership label and the environment identifier -- AND the scripts remove only that environment's namespaces +- THEN the scripts remove only that environment's platform project and `-keycloak` project, or the HyperShell resources in them - AND the other environment stays intact #### Scenario: Deployment refuses a foreign namespace @@ -295,6 +350,15 @@ SHALL refuse a mismatch and report it. - THEN the command refuses to adopt the namespace - AND the command deploys nothing into, and deletes nothing in, that namespace +#### Scenario: Existing unlabeled project is used without labeling + +- GIVEN the current oc project already exists and is not HyperShell-labeled +- AND the developer cannot patch namespace objects +- WHEN the developer runs `make openshift-up` +- THEN the command deploys into that project without prompting +- AND the command does not stop because namespace labeling is forbidden +- AND `make openshift-down` for that project deletes the platform and `-keycloak` projects, or the HyperShell resources in them + ### Requirement: Keycloak Namespace The deployment SHALL place Keycloak in its own namespace, separate from the other @@ -315,8 +379,10 @@ Keycloak in its own namespace. Together, the platform namespace and its `-keycloak` namespace form the deployment's namespace group. The two namespaces SHALL share one lifecycle: the -scripts create them together, and `make openshift-down` (for local development) or -the release step (for CI) removes them together. +scripts create missing ones with `oc new-project`, apply Keycloak while that +project is selected, switch back to the platform project for the remaining +components, and `make openshift-down` / `make openshift-teardown` (for local development) or the release step +(for CI) removes them together. The OpenShift OIDC configuration SHALL derive from the Keycloak Route in the `-keycloak` namespace through one hostname formula: the driver reads the Keycloak @@ -339,15 +405,33 @@ their own namespaces is out of scope here and belongs to a separate spec. #### Scenario: Keycloak runs in its own namespace - GIVEN a developer runs `make openshift-up` with `OPENSHIFT_NAMESPACE=alice` +- AND the `alice-keycloak` project does not exist - WHEN the deployment is ready -- THEN Keycloak runs in the `alice-keycloak` namespace +- THEN the scripts have created `alice-keycloak` with `oc new-project` +- AND Keycloak runs in the `alice-keycloak` namespace - AND every other HyperShell component runs in the `alice` namespace +- AND the current oc project is `alice` after Keycloak is applied - AND the OIDC issuer points at the Keycloak route in `alice-keycloak` +#### Scenario: Platform workloads can reach Keycloak across the namespace group + +`oc new-project` installs default-deny Ingress NetworkPolicies in each +project (same-namespace pods and the `openshift-ingress` namespace). Keycloak +runs in `${OPENSHIFT_NAMESPACE}-keycloak`, so platform pods cannot reach it +until an additional policy allows that traffic. Without it, the API server +hangs while loading JWKS and the rollout never completes. + +- GIVEN a developer runs `make openshift-up` with `OPENSHIFT_NAMESPACE=alice` +- AND `oc new-project` has created default-deny Ingress policies in `alice-keycloak` +- WHEN the deployment is ready +- THEN a NetworkPolicy in `alice-keycloak` allows TCP/8080 from the `alice` namespace +- AND the API server can load JWKS from `keycloak-service.alice-keycloak` +- AND the control plane can reach the Keycloak Admin API on that Service + #### Scenario: The namespace group shares one lifecycle - GIVEN a deployment has a platform namespace and its `-keycloak` namespace -- WHEN the deployment is removed +- WHEN the developer runs `make openshift-down` or `make openshift-teardown` - THEN both namespaces are removed together - AND the `-keycloak` namespace is not left behind From 8d8a0f487e6405f3cbc54d705b93e3f35afb0c7e Mon Sep 17 00:00:00 2001 From: Kyle Squizzato Date: Mon, 31 Aug 2026 12:25:31 -0700 Subject: [PATCH 02/14] fix(e2e-openshift): create/patch a hypershell-e2e ClusterRole Assisted-by: Cursor Grok 4.6 Signed-off-by: Kyle Squizzato --- DEVELOPMENT.md | 10 +- scripts/cluster/drivers/openshift.sh | 171 ++++++++++--------- scripts/cluster/lib_test.sh | 138 ++++++++++++++- scripts/cluster/rewrite-namespaces.py | 103 ++++++++++- skills/RECONCILE.md | 2 +- specs/platform/openshift-development.spec.md | 52 ++++-- 6 files changed, 360 insertions(+), 116 deletions(-) diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md index 42d045d3..d8a3a2ee 100644 --- a/DEVELOPMENT.md +++ b/DEVELOPMENT.md @@ -333,13 +333,21 @@ Service. This renders `kustomize build deploy/openshift/`, maps `hypershell-system` to that platform namespace and `keycloak` to `${platform}-keycloak`, applies the +required cluster-scoped RBAC (shared ClusterRole `hypershell-e2e`, per-namespace +ClusterRoleBindings, and the privileged SCC RoleBinding), applies the manifests (with prune scoped to this environment), registers the web-console Route as the Keycloak `hypershell-frontend` redirect URI, seeds a ManagedCluster, GatewayRelease, ManagedDatabase, and Gateway from this machine against the API and Keycloak Routes (the API server image has no `curl`), and prints the API, web-console, and Keycloak Routes. The gateway base domain is read from the shared Gateway's listener hostname, not from -`GATEWAY_API_BASE_DOMAIN`. +`GATEWAY_API_BASE_DOMAIN`. ClusterRole `hypershell-e2e` is shared by every e2e +environment on the cluster (`oc apply` creates or patches it). ClusterRoleBindings +are prefixed per project so they do not touch stage's `hypershell-controller`. +Cluster-scoped RBAC is not optional: without it the controller cannot create +gateway namespaces or sandboxes, so `make openshift-up` fails if `hypershell-e2e` +is missing and cannot be created. `make openshift-down` deletes this +environment's prefixed ClusterRoleBindings and leaves `hypershell-e2e` in place. `make openshift-down` and `make openshift-teardown` are the same command. There is no OpenShift cluster to destroy. Both delete the platform project diff --git a/scripts/cluster/drivers/openshift.sh b/scripts/cluster/drivers/openshift.sh index 5e1ed152..d1d81aa1 100755 --- a/scripts/cluster/drivers/openshift.sh +++ b/scripts/cluster/drivers/openshift.sh @@ -266,16 +266,63 @@ check_infrastructure() { info "Gateway base domain: ${GATEWAY_API_BASE_DOMAIN} (from ${gw_ns}/${gw_name} listener)" } -apply_bind_grant() { - # Privileged local-dev actor: grant this environment's controller SA the - # cluster-scoped bind on the privileged SCC. Typical developers cannot - # create ClusterRoleBindings; skip rather than fail the whole bring-up. +fail_required_cluster_rbac() { + local err="$1" + error "Cluster-scoped RBAC is required to provision gateways and sandboxes." + error "${err}" + error "An administrator must apply ClusterRole hypershell-e2e once on this cluster, plus this namespace's ClusterRoleBindings and RoleBinding hypershell-sandbox-scc." + error "Do not change ClusterRole hypershell-controller; that belongs to stage." + exit 1 +} + +assert_expected_cluster_scoped() { + local rendered="$1" + local bad + if ! bad="$(python3 -c ' +from importlib.machinery import SourceFileLoader +from pathlib import Path +import sys +rw = SourceFileLoader("rewrite", sys.argv[1]).load_module() +bad = rw.unprefixed_cluster_scoped(Path(sys.argv[2]).read_text(), sys.argv[3]) +if bad: + print("\n".join(bad)) + raise SystemExit(1) +' "${CLUSTER_SCRIPT_DIR}/rewrite-namespaces.py" "${rendered}" "${OPENSHIFT_NAMESPACE}-")"; then + error "Refusing cluster-scoped names that would collide with stage:" + error "${bad}" + exit 1 + fi +} + +ensure_e2e_cluster_role() { + # One shared ClusterRole for every e2e environment. oc apply creates or + # patches when the rules differ. If this user cannot apply it, reuse it + # when it already exists; fail if it is missing. + local rendered="$1" + local err + assert_expected_cluster_scoped "${rendered}" + if err="$(oc_cli apply -f "${rendered}" 2>&1)"; then + info "ClusterRole hypershell-e2e applied" + printf '%s\n' "${err}" + rm -f "${rendered}" + return 0 + fi + rm -f "${rendered}" + if grep -qi 'forbidden' <<<"${err}" && oc_cli get clusterrole hypershell-e2e >/dev/null 2>&1; then + warn "Cannot update ClusterRole hypershell-e2e (forbidden); using the existing role." + return 0 + fi + fail_required_cluster_rbac "${err}" +} + +apply_sandbox_scc() { local err - if err="$(oc_cli apply -f - 2>&1 <&1 <&1)"; then + rm -f "${bindings}" + fail_required_cluster_rbac "${err}" + fi + printf '%s\n' "${err}" + rm -f "${bindings}" + + apply_sandbox_scc } create_bootstrap_secrets() { @@ -440,29 +510,6 @@ apply_rendered_overlay() { rm -f "${rendered}" } -apply_optional() { - local rendered="$1" - local what="$2" - if [[ ! -s "${rendered}" ]]; then - rm -f "${rendered}" - return 0 - fi - local err - if err="$(oc_cli apply -f "${rendered}" 2>&1)"; then - printf '%s\n' "${err}" - rm -f "${rendered}" - return 0 - fi - rm -f "${rendered}" - if grep -qi 'forbidden' <<<"${err}"; then - warn "Skipping ${what}: the current user cannot apply cluster-scoped or privileged RBAC." - warn "An administrator must grant this namespace's hypershell-controller SA the shared ClusterRole and privileged SCC bind." - return 0 - fi - error "${err}" - return 1 -} - apply_postgres_fallback() { info "CNPG operator is not installed; deploying bundled PostgreSQL Deployment" local rendered @@ -484,9 +531,8 @@ apply_postgres_fallback() { } configure_postgres_fallback_ssl() { - oc_cli set env deployment/hypershell-api-server -n "${OPENSHIFT_NAMESPACE}" -c migrate \ - DB_SSLMODE=disable >/dev/null - oc_cli set env deployment/hypershell-api-server -n "${OPENSHIFT_NAMESPACE}" -c api-server \ + # migrate is an initContainer; omit -c so oc sets both init and app containers. + oc_cli set env deployment/hypershell-api-server -n "${OPENSHIFT_NAMESPACE}" \ DB_SSLMODE=disable >/dev/null } @@ -529,50 +575,9 @@ apply_overlay() { configure_postgres_fallback_ssl fi - info "Applying optional cluster-scoped RBAC (best-effort)..." - if rendered="$(render_openshift_manifests --only-namespace __cluster__ --include-cluster-scoped)"; then - apply_optional "${rendered}" "ClusterRole / ClusterRoleBinding" - fi - apply_sandbox_scc_best_effort - success "Overlay applied" } -apply_sandbox_scc_best_effort() { - local err - if err="$(oc_cli apply -n "${OPENSHIFT_NAMESPACE}" -f - 2>&1 </dev/null 2>&1 || true oc_cli delete clusterrolebinding "${OPENSHIFT_NAMESPACE}-hypershell-controller" --ignore-not-found >/dev/null 2>&1 || true diff --git a/scripts/cluster/lib_test.sh b/scripts/cluster/lib_test.sh index 2ab9d53e..a53d6764 100755 --- a/scripts/cluster/lib_test.sh +++ b/scripts/cluster/lib_test.sh @@ -73,18 +73,48 @@ if grep -E 'deletion started|removal started' "${SCRIPT_DIR}/drivers/openshift.s else PASS=$((PASS + 1)) fi -if grep -A40 '^cluster_down()' "${SCRIPT_DIR}/drivers/openshift.sh" | grep -q 'OPENSHIFT_KEYCLOAK_NAMESPACE'; then +if grep -A50 '^cluster_down()' "${SCRIPT_DIR}/drivers/openshift.sh" | grep -q 'OPENSHIFT_KEYCLOAK_NAMESPACE'; then PASS=$((PASS + 1)) else FAIL=$((FAIL + 1)) echo 'FAIL: OpenShift cluster_down does not remove the Keycloak namespace' fi -if grep -E 'oc_cli exec.*curl|oc exec.*curl' "${SCRIPT_DIR}/drivers/openshift.sh" >/dev/null; then +if grep -A50 '^cluster_down()' "${SCRIPT_DIR}/drivers/openshift.sh" | grep -q 'delete clusterrole '; then FAIL=$((FAIL + 1)) - echo 'FAIL: OpenShift driver still oc execs curl in a pod' + echo 'FAIL: OpenShift cluster_down deletes a ClusterRole (must not remove shared hypershell-e2e or stage)' else PASS=$((PASS + 1)) fi +if grep -A50 '^cluster_down()' "${SCRIPT_DIR}/drivers/openshift.sh" | grep -q 'clusterrolebinding "${OPENSHIFT_NAMESPACE}-hypershell-controller"'; then + PASS=$((PASS + 1)) +else + FAIL=$((FAIL + 1)) + echo 'FAIL: OpenShift cluster_down does not delete this environment'\''s prefixed ClusterRoleBinding' +fi +if grep -E 'delete clusterrole(binding)? "hypershell-controller' "${SCRIPT_DIR}/drivers/openshift.sh"; then + FAIL=$((FAIL + 1)) + echo 'FAIL: OpenShift down deletes unprefixed cluster-scoped names (would hit stage)' +else + PASS=$((PASS + 1)) +fi +if grep -q 'optional cluster-scoped RBAC (best-effort)' "${SCRIPT_DIR}/drivers/openshift.sh"; then + FAIL=$((FAIL + 1)) + echo 'FAIL: OpenShift still treats cluster-scoped RBAC as best-effort' +else + PASS=$((PASS + 1)) +fi +if grep -q 'Skipping ClusterRole' "${SCRIPT_DIR}/drivers/openshift.sh"; then + FAIL=$((FAIL + 1)) + echo 'FAIL: OpenShift still skips ClusterRole apply' +else + PASS=$((PASS + 1)) +fi +if grep -A80 '^apply_required_cluster_rbac()' "${SCRIPT_DIR}/drivers/openshift.sh" | grep -q 'fail_required_cluster_rbac'; then + PASS=$((PASS + 1)) +else + FAIL=$((FAIL + 1)) + echo 'FAIL: apply_required_cluster_rbac does not fail when cluster RBAC cannot be applied' +fi assert_eq "alice-keycloak" "$(keycloak_namespace_for alice)" "keycloak namespace suffix" assert_ok "derived keycloak ns fits 63" validate_rfc1123_label "$(keycloak_namespace_for "$(printf 'a%.0s' {1..54})")" 63 assert_eq "openshell.example.com" "$(gateway_base_domain_from_hostname '*.openshell.example.com')" "strip wildcard listener hostname" @@ -246,19 +276,25 @@ if printf '%s' "${rewritten}" | grep -q 'controllerroleRef'; then else PASS=$((PASS + 1)) fi -# ClusterRole name and roleRef.name must stay unprefixed +# Shared ClusterRole hypershell-e2e; ClusterRoleBinding roleRef points at it if printf '%s' "${rewritten}" | grep -q 'kind: ClusterRole' \ - && printf '%s' "${rewritten}" | awk '/kind: ClusterRole$/{p=1} p&&/name:/{print; exit}' | grep -q 'name: hypershell-controller'; then + && printf '%s' "${rewritten}" | awk '/kind: ClusterRole$/{p=1} p&&/^ name:/{print; exit}' | grep -q 'name: hypershell-e2e'; then PASS=$((PASS + 1)) else FAIL=$((FAIL + 1)) - echo 'FAIL: ClusterRole name should stay hypershell-controller' + echo 'FAIL: ClusterRole name should be hypershell-e2e' fi -if printf '%s' "${rewritten}" | grep -A4 'roleRef:' | grep -q 'name: hypershell-controller'; then +if printf '%s' "${rewritten}" | grep -A4 'roleRef:' | grep -q 'name: hypershell-e2e'; then PASS=$((PASS + 1)) else FAIL=$((FAIL + 1)) - echo 'FAIL: roleRef.name should stay hypershell-controller' + echo 'FAIL: ClusterRoleBinding roleRef.name should be hypershell-e2e' +fi +if printf '%s' "${rewritten}" | awk '/kind: ClusterRole$/{p=1} p&&/^ name:/{print; exit}' | grep -qx ' name: hypershell-controller'; then + FAIL=$((FAIL + 1)) + echo 'FAIL: ClusterRole metadata.name was left as hypershell-controller (would collide with stage)' +else + PASS=$((PASS + 1)) fi if printf '%s' "${rewritten}" | grep -q 'value: alice'; then PASS=$((PASS + 1)) @@ -372,6 +408,49 @@ else FAIL=$((FAIL + 1)) echo 'FAIL: roleRef was not kept as its own line' fi +if printf '%s' "${crb_out}" | grep -A3 '^roleRef:' | grep -qx ' name: hypershell-e2e'; then + PASS=$((PASS + 1)) +else + FAIL=$((FAIL + 1)) + echo 'FAIL: ClusterRoleBinding roleRef.name should be hypershell-e2e' + printf '%s\n' "${crb_out}" +fi + +sys_crb="$(mktemp)" +cat > "${sys_crb}" <<'EOF' +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: hypershell-sandbox-scc +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: system:openshift:scc:privileged +subjects: +- kind: ServiceAccount + name: openshell-gateway-sandbox + namespace: hypershell-system +EOF +sys_out="$(python3 "${SCRIPT_DIR}/rewrite-namespaces.py" \ + --platform-namespace alice \ + --keycloak-namespace alice-keycloak \ + --omit-namespaces \ + --only-namespace __cluster__ \ + --include-cluster-scoped < "${sys_crb}")" +rm -f "${sys_crb}" +if printf '%s' "${sys_out}" | grep -qx ' name: alice-hypershell-sandbox-scc'; then + PASS=$((PASS + 1)) +else + FAIL=$((FAIL + 1)) + echo 'FAIL: ClusterRoleBinding to a built-in ClusterRole was not name-prefixed' +fi +if printf '%s' "${sys_out}" | grep -A3 '^roleRef:' | grep -qx ' name: system:openshift:scc:privileged'; then + PASS=$((PASS + 1)) +else + FAIL=$((FAIL + 1)) + echo 'FAIL: built-in system: ClusterRole roleRef was renamed' + printf '%s\n' "${sys_out}" +fi omit_cr="$(python3 "${SCRIPT_DIR}/rewrite-namespaces.py" \ --platform-namespace alice \ --keycloak-namespace alice-keycloak \ @@ -386,6 +465,26 @@ else PASS=$((PASS + 1)) fi +only_cr="$(python3 "${SCRIPT_DIR}/rewrite-namespaces.py" \ + --platform-namespace alice \ + --keycloak-namespace alice-keycloak \ + --omit-namespaces \ + --only-namespace __cluster__ \ + --include-cluster-scoped \ + --only-kinds ClusterRole <<<"${rewritten}")" +if printf '%s' "${only_cr}" | grep -q 'kind: ClusterRole'; then + PASS=$((PASS + 1)) +else + FAIL=$((FAIL + 1)) + echo 'FAIL: --only-kinds ClusterRole dropped ClusterRole' +fi +if printf '%s' "${only_cr}" | grep -q 'kind: ClusterRoleBinding'; then + FAIL=$((FAIL + 1)) + echo 'FAIL: --only-kinds ClusterRole kept ClusterRoleBinding' +else + PASS=$((PASS + 1)) +fi + stripped="$(python3 "${SCRIPT_DIR}/rewrite-namespaces.py" \ --platform-namespace alice \ --keycloak-namespace alice-keycloak \ @@ -451,6 +550,29 @@ if command -v kustomize >/dev/null 2>&1; then FAIL=$((FAIL + 1)) echo 'FAIL: rewritten overlay still contains hypershell-system' fi + unprefixed="$(python3 -c ' +from importlib.machinery import SourceFileLoader +from pathlib import Path +import sys +rw = SourceFileLoader("rewrite", sys.argv[1]).load_module() +bad = rw.unprefixed_cluster_scoped(Path(sys.argv[2]).read_text(), "alice-") +if bad: + print("\n".join(bad)) + raise SystemExit(1) +' "${SCRIPT_DIR}/rewrite-namespaces.py" "${os_out}")" || true + if [[ -z "${unprefixed}" ]]; then + PASS=$((PASS + 1)) + else + FAIL=$((FAIL + 1)) + echo "FAIL: rewritten overlay has unprefixed cluster-scoped names:" + printf '%s\n' "${unprefixed}" + fi + if grep -q 'name: system:openshift:scc:privileged' "${os_out}"; then + PASS=$((PASS + 1)) + else + FAIL=$((FAIL + 1)) + echo 'FAIL: namespaced SCC RoleBinding lost built-in system: roleRef' + fi else FAIL=$((FAIL + 1)) echo 'FAIL: rewritten overlay render' diff --git a/scripts/cluster/rewrite-namespaces.py b/scripts/cluster/rewrite-namespaces.py index e40ba0fe..e29d4054 100755 --- a/scripts/cluster/rewrite-namespaces.py +++ b/scripts/cluster/rewrite-namespaces.py @@ -3,8 +3,12 @@ Maps the overlay's platform namespace (hypershell-system) to OPENSHIFT_NAMESPACE and the bundled Keycloak namespace (keycloak) to ${OPENSHIFT_NAMESPACE}-keycloak. -ClusterRoleBindings are renamed so two environments on one cluster do not share -a binding name. ClusterRoles stay shared. + +ClusterRoles from the overlay are merged into one shared ClusterRole named +hypershell-e2e so every e2e environment reuses the same rules and does not patch +stage's hypershell-controller. ClusterRoleBindings stay prefixed per platform +namespace; their roleRef points at hypershell-e2e unless it is a built-in +system: ClusterRole. """ from __future__ import annotations @@ -14,6 +18,8 @@ PLATFORM_NS = "hypershell-system" KEYCLOAK_NS = "keycloak" +E2E_CLUSTER_ROLE = "hypershell-e2e" +CLUSTER_SCOPED_KINDS = frozenset({"ClusterRole", "ClusterRoleBinding"}) def split_docs(text: str) -> list[str]: @@ -34,8 +40,8 @@ def kind_of(doc: str) -> str: return match.group(1) if match else "" -def prefix_cluster_role_binding_name(doc: str, prefix: str) -> str: - """Prefix metadata.name only; leave roleRef.name unchanged.""" +def prefix_metadata_name(doc: str, prefix: str) -> str: + """Prefix metadata.name only; do not touch roleRef or subjects.""" parts = re.split(r"^(roleRef:|subjects:)", doc, maxsplit=1, flags=re.M) head = parts[0] rest = "".join(parts[1:]) if len(parts) > 1 else "" @@ -52,6 +58,23 @@ def repl(match: re.Match[str]) -> str: return head + rest +def set_e2e_cluster_role_ref(doc: str) -> str: + """Point roleRef at hypershell-e2e unless it is a built-in system: ClusterRole.""" + parts = re.split(r"^(roleRef:)", doc, maxsplit=1, flags=re.M) + if len(parts) < 3: + return doc + head, marker, rest = parts[0], parts[1], parts[2] + + def repl(match: re.Match[str]) -> str: + name = match.group(2) + if name.startswith("system:") or name == E2E_CLUSTER_ROLE: + return match.group(0) + return f"{match.group(1)}{E2E_CLUSTER_ROLE}" + + rest = re.sub(r"^( name:\s*)(\S+)[^\S\n]*$", repl, rest, count=1, flags=re.M) + return head + marker + rest + + def rewrite_doc(doc: str, platform_ns: str, keycloak_ns: str) -> str: kind = kind_of(doc) rewritten = doc.replace(PLATFORM_NS, platform_ns) @@ -69,7 +92,8 @@ def rewrite_doc(doc: str, platform_ns: str, keycloak_ns: str) -> str: flags=re.M, ) if kind == "ClusterRoleBinding": - rewritten = prefix_cluster_role_binding_name(rewritten, f"{platform_ns}-") + rewritten = prefix_metadata_name(rewritten, f"{platform_ns}-") + rewritten = set_e2e_cluster_role_ref(rewritten) return rewritten @@ -90,6 +114,62 @@ def metadata_name(doc: str) -> str | None: return name.group(1) if name else None +def unprefixed_cluster_scoped(text: str, prefix: str) -> list[str]: + """Return cluster-scoped ids that would collide with stage or another instance. + + ClusterRole must be the shared hypershell-e2e name. ClusterRoleBinding names + must start with the platform-namespace prefix. + """ + bad: list[str] = [] + for doc in split_docs(text): + kind = kind_of(doc) + name = metadata_name(doc) or "" + if kind == "ClusterRole": + if name != E2E_CLUSTER_ROLE: + bad.append(f"{kind}/{name or '(missing name)'}") + elif kind == "ClusterRoleBinding": + if not name.startswith(prefix): + bad.append(f"{kind}/{name or '(missing name)'}") + return bad + + +def cluster_role_rules_block(doc: str) -> str: + match = re.search(r"^rules:\n(.*)\Z", doc, re.S | re.M) + if not match: + return "" + return match.group(1).rstrip("\n") + + +def merge_e2e_cluster_role(docs: list[str]) -> list[str]: + """Collapse overlay ClusterRoles into one shared hypershell-e2e ClusterRole.""" + others: list[str] = [] + rule_blocks: list[str] = [] + for doc in docs: + if kind_of(doc) != "ClusterRole": + others.append(doc) + continue + rules = cluster_role_rules_block(doc) + if rules: + rule_blocks.append(rules) + if not rule_blocks and not any(kind_of(d) == "ClusterRole" for d in docs): + return others + rules_yaml = "\n".join(rule_blocks) if rule_blocks else " []" + merged = ( + "apiVersion: rbac.authorization.k8s.io/v1\n" + "kind: ClusterRole\n" + "metadata:\n" + f" name: {E2E_CLUSTER_ROLE}\n" + " labels:\n" + " app.kubernetes.io/name: hypershell\n" + " app.kubernetes.io/component: e2e\n" + " app.kubernetes.io/part-of: hypershell\n" + ' hypershell.redhat.io/shared-e2e: "true"\n' + "rules:\n" + f"{rules_yaml}\n" + ) + return [merged] + others + + def keep_doc( doc: str, *, @@ -98,12 +178,15 @@ def keep_doc( include_cluster_scoped: bool, omit_kinds: set[str] | None = None, omit_names: set[str] | None = None, + only_kinds: set[str] | None = None, ) -> bool: if not doc.strip(): return False kind = kind_of(doc) if omit_namespaces and kind == "Namespace": return False + if only_kinds and kind not in only_kinds: + return False if omit_kinds and kind in omit_kinds: return False if omit_names: @@ -127,8 +210,10 @@ def rewrite( include_cluster_scoped: bool = False, omit_kinds: set[str] | None = None, omit_names: set[str] | None = None, + only_kinds: set[str] | None = None, ) -> str: docs = [rewrite_doc(doc, platform_ns, keycloak_ns) for doc in split_docs(text)] + docs = merge_e2e_cluster_role(docs) docs = [ doc for doc in docs @@ -139,6 +224,7 @@ def rewrite( include_cluster_scoped=include_cluster_scoped, omit_kinds=omit_kinds, omit_names=omit_names, + only_kinds=only_kinds, ) ] rendered = "\n---\n".join(docs) @@ -181,6 +267,11 @@ def main() -> int: default="", help="Comma-separated kinds to drop (e.g. ClusterRole,ClusterRoleBinding,Cluster).", ) + parser.add_argument( + "--only-kinds", + default="", + help="Comma-separated kinds to keep (e.g. ClusterRole,ClusterRoleBinding).", + ) parser.add_argument( "--omit-names", default="", @@ -194,6 +285,7 @@ def main() -> int: args = parser.parse_args() omit_kinds = {k for k in args.omit_kinds.split(",") if k} omit_names = {n for n in args.omit_names.split(",") if n} + only_kinds = {k for k in args.only_kinds.split(",") if k} rendered = rewrite( sys.stdin.read(), args.platform_namespace, @@ -203,6 +295,7 @@ def main() -> int: include_cluster_scoped=args.include_cluster_scoped, omit_kinds=omit_kinds or None, omit_names=omit_names or None, + only_kinds=only_kinds or None, ) if args.strip_openshift_uids: rendered = strip_openshift_fixed_uids(rendered) diff --git a/skills/RECONCILE.md b/skills/RECONCILE.md index 76c72cb9..e2a62e07 100644 --- a/skills/RECONCILE.md +++ b/skills/RECONCILE.md @@ -136,7 +136,7 @@ Layer 7: web-console/architecture (depends on data-model, security, UI | OS-10 | Blessed OpenShift Overlay | Partial | Namespace parameterization, Routes, SCC RoleBindings; gateway base domain discovered from the shared Gateway listener (not `GATEWAY_API_BASE_DOMAIN`). Drift-check CI job deferred. | `deploy/openshift/`, `rewrite-namespaces.py` | OS-W2 | | OS-11 | OpenShift CI Workflow Shape | Missing | Intentionally deferred: not local-dev lifecycle | - | Future | | OS-12 | Cluster Infrastructure Prerequisites | Present | `make openshift-up` fails fast when the shared Gateway is missing or not Programmed. GatewayClass is cluster-scoped and not GET-checked (developers typically cannot read it). | `drivers/openshift.sh` `check_infrastructure` | OS-W2 | -| OS-13 | Cluster-Scoped Permissions + SCC/RBAC posture | Present | Namespace-scoped SCC *use* RoleBindings; per-namespace `bind` ClusterRoleBinding; overlay keeps `RBAC_ENFORCE=true`. E2E RBAC scenarios stay with the e2e suite. | `deploy/openshift/scc.yaml`, `drivers/openshift.sh` | OS-W2 | +| OS-13 | Cluster-Scoped Permissions + SCC/RBAC posture | Present | Shared ClusterRole `hypershell-e2e` (`oc apply` create-or-patch); per-namespace prefixed ClusterRoleBindings; fail if the role is missing and cannot be created. Overlay keeps `RBAC_ENFORCE=true`. | `rewrite-namespaces.py`, `drivers/openshift.sh` `ensure_e2e_cluster_role` | OS-W2 | Local-dev lifecycle (`make openshift-up` / `down` / component swaps) is implemented. E2E driver completion beyond the OS-W1 manual slice, legacy `pr-test` consolidation, ephemeral CI, access handoff, overlay drift CI, and the OpenShift e2e workflow remain out of scope for this wave. diff --git a/specs/platform/openshift-development.spec.md b/specs/platform/openshift-development.spec.md index a308e622..bdfad5d3 100644 --- a/specs/platform/openshift-development.spec.md +++ b/specs/platform/openshift-development.spec.md @@ -334,6 +334,8 @@ the projects. The command SHALL NOT require namespace labels in order to delete. - THEN each deployment runs in its own namespace group - AND each deployment has distinct gateway hostnames - AND neither deployment changes, conflicts with, or interacts with the other +- AND each environment's ClusterRoleBindings are prefixed with that environment's platform namespace and refer to ClusterRole `hypershell-e2e` +- AND neither environment patches unprefixed names such as `hypershell-controller` that another instance (for example stage) already owns #### Scenario: Namespace cleanup removes only one deployment @@ -935,33 +937,47 @@ controller needs so that it can itself create the per-namespace privileged RoleB for a sandbox at runtime. `bind` on `clusterroles` is a cluster-scoped permission. A namespace-scoped RoleBinding CANNOT grant it, and a single pre-created ClusterRoleBinding with a fixed service-account subject cannot cover the controller -service account of an unknown future ephemeral namespace. Therefore a privileged -actor SHALL grant each ephemeral controller service account the cluster-scoped `bind` -on the privileged SCC. For example, the namespace-provisioning step (the -ephemeral-namespace operator, or an equivalent cluster-privileged step that creates -the namespace and its service accounts) SHALL, at namespace-create time, add that -controller service account as a subject of a `bind` ClusterRoleBinding or create a -per-namespace ClusterRoleBinding for it. Alternatively, a privileged CI deployer that -is not the in-namespace controller SHALL create the per-namespace privileged -RoleBindings for the sandbox, so that the in-namespace controller never needs `bind`. +service account of an unknown future ephemeral namespace. + +For local-dev, `make openshift-up` is that privileged actor. It SHALL apply one +shared ClusterRole named `hypershell-e2e` (create or patch when the rules differ), +per-environment ClusterRoleBindings whose names are prefixed with the platform +namespace and whose `roleRef` is `hypershell-e2e`, and the privileged SCC RoleBinding +`hypershell-sandbox-scc`. Gateways and sandboxes cannot run without those grants, so +the command SHALL fail if the ClusterRole is missing and cannot be created, or if the +per-environment bindings cannot be applied. If `hypershell-e2e` already exists and +the current user cannot patch it, the command SHALL reuse the existing ClusterRole. +It SHALL NOT create, patch, or delete ClusterRole `hypershell-controller` or +ClusterRoleBinding `hypershell-controller`; those names belong to other instances +that share the cluster, such as stage. Built-in ClusterRoles whose names start with +`system:` SHALL NOT be renamed. `make openshift-down` SHALL delete only this +environment's prefixed ClusterRoleBindings, not ClusterRole `hypershell-e2e`. The deployment into the ephemeral namespace SHALL NOT attempt to grant the cluster-scoped `bind` through a namespace-scoped RoleBinding, and SHALL NOT assume the namespace-scoped RoleBindings alone let the controller bind the privileged SCC per -namespace. The cluster-scoped `bind` ClusterRole and the grant of `bind` to each -ephemeral controller service account SHALL be pre-created or provisioned by the -privileged actor, as above, before the controller reconciles a sandbox. +namespace. #### Scenario: Ephemeral namespace has the permissions the overlay needs -- GIVEN the target cluster pre-creates the cluster-scoped `bind` ClusterRole and its binding -- AND a privileged actor grants the ephemeral controller service account the cluster-scoped `bind` on the privileged SCC at namespace-create time -- WHEN the workflow deploys the OpenShift overlay into an ephemeral namespace -- THEN the deployment creates only namespace-scoped RoleBindings that attach the - controller and sandbox service accounts to the built-in SCC ClusterRoles for SCC use -- AND the controller can create the per-namespace privileged RoleBinding for a sandbox because the privileged actor granted it `bind` +- GIVEN ClusterRole `hypershell-e2e` can be applied, or already exists +- AND the developer can apply per-namespace ClusterRoleBindings and the privileged SCC RoleBinding +- WHEN the developer runs `make openshift-up` +- THEN the command applies ClusterRole `hypershell-e2e` (creating it or patching it when it differs) +- AND applies ClusterRoleBindings prefixed with the platform namespace whose `roleRef` is `hypershell-e2e` +- AND the command does not patch ClusterRole `hypershell-controller` or ClusterRoleBinding `hypershell-controller` +- AND applies RoleBinding `hypershell-sandbox-scc` for the sandbox service account +- AND the controller can create the per-namespace privileged RoleBinding for a sandbox because it was granted `bind` - AND the deployment does not attempt to grant the cluster-scoped `bind` through a namespace-scoped RoleBinding +#### Scenario: Cluster-scoped RBAC cannot be applied + +- GIVEN the current user cannot create ClusterRoleBindings or grant the privileged SCC +- WHEN the developer runs `make openshift-up` +- THEN the command fails +- AND the error states that cluster-scoped RBAC is required to provision gateways and sandboxes +- AND the command does not report the overlay as applied + ### Requirement: OpenShift Security Context and RBAC Parity The OpenShift deployment SHALL keep the security posture that the OpenShift overlay From 65ca995a7448e27e83e34159a7ad810d8f58a3f7 Mon Sep 17 00:00:00 2001 From: Kyle Squizzato Date: Tue, 1 Sep 2026 07:31:52 -0700 Subject: [PATCH 03/14] fix(e2e-openshift): prefix cluster RBAC with ${ns}-dev- A shared hypershell-e2e ClusterRole would let one environment's overlay patch break every other e2e env. Prefix ClusterRoles and ClusterRoleBindings per project instead, and delete only this env's objects on teardown. Assisted-by: Cursor Grok 4.6 Signed-off-by: Kyle Squizzato --- DEVELOPMENT.md | 16 ++-- scripts/cluster/drivers/openshift.sh | 61 ++++---------- scripts/cluster/lib_test.sh | 36 ++++----- scripts/cluster/rewrite-namespaces.py | 83 ++++++-------------- skills/RECONCILE.md | 2 +- specs/platform/openshift-development.spec.md | 24 +++--- 6 files changed, 75 insertions(+), 147 deletions(-) diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md index d8a3a2ee..ca1aad37 100644 --- a/DEVELOPMENT.md +++ b/DEVELOPMENT.md @@ -333,21 +333,19 @@ Service. This renders `kustomize build deploy/openshift/`, maps `hypershell-system` to that platform namespace and `keycloak` to `${platform}-keycloak`, applies the -required cluster-scoped RBAC (shared ClusterRole `hypershell-e2e`, per-namespace -ClusterRoleBindings, and the privileged SCC RoleBinding), applies the +required cluster-scoped RBAC (ClusterRoles and ClusterRoleBindings named +`${namespace}-dev-*`, plus the privileged SCC RoleBinding), applies the manifests (with prune scoped to this environment), registers the web-console Route as the Keycloak `hypershell-frontend` redirect URI, seeds a ManagedCluster, GatewayRelease, ManagedDatabase, and Gateway from this machine against the API and Keycloak Routes (the API server image has no `curl`), and prints the API, web-console, and Keycloak Routes. The gateway base domain is read from the shared Gateway's listener hostname, not from -`GATEWAY_API_BASE_DOMAIN`. ClusterRole `hypershell-e2e` is shared by every e2e -environment on the cluster (`oc apply` creates or patches it). ClusterRoleBindings -are prefixed per project so they do not touch stage's `hypershell-controller`. -Cluster-scoped RBAC is not optional: without it the controller cannot create -gateway namespaces or sandboxes, so `make openshift-up` fails if `hypershell-e2e` -is missing and cannot be created. `make openshift-down` deletes this -environment's prefixed ClusterRoleBindings and leaves `hypershell-e2e` in place. +`GATEWAY_API_BASE_DOMAIN`. Cluster-scoped names use the `${namespace}-dev-` +prefix so an environment does not patch stage's `hypershell-controller`. Applying +that RBAC is required: `make openshift-up` fails if the current user cannot create +those ClusterRoles and ClusterRoleBindings. `make openshift-down` deletes this +environment's `${namespace}-dev-*` ClusterRoles and ClusterRoleBindings. `make openshift-down` and `make openshift-teardown` are the same command. There is no OpenShift cluster to destroy. Both delete the platform project diff --git a/scripts/cluster/drivers/openshift.sh b/scripts/cluster/drivers/openshift.sh index d1d81aa1..c6dfd48b 100755 --- a/scripts/cluster/drivers/openshift.sh +++ b/scripts/cluster/drivers/openshift.sh @@ -270,13 +270,14 @@ fail_required_cluster_rbac() { local err="$1" error "Cluster-scoped RBAC is required to provision gateways and sandboxes." error "${err}" - error "An administrator must apply ClusterRole hypershell-e2e once on this cluster, plus this namespace's ClusterRoleBindings and RoleBinding hypershell-sandbox-scc." - error "Do not change ClusterRole hypershell-controller; that belongs to stage." + error "The current user must be able to create ClusterRole and ClusterRoleBinding (${OPENSHIFT_NAMESPACE}-dev-*)." + error "Do not apply unprefixed hypershell-controller; that belongs to stage." exit 1 } assert_expected_cluster_scoped() { local rendered="$1" + local prefix="${OPENSHIFT_NAMESPACE}-dev-" local bad if ! bad="$(python3 -c ' from importlib.machinery import SourceFileLoader @@ -287,34 +288,13 @@ bad = rw.unprefixed_cluster_scoped(Path(sys.argv[2]).read_text(), sys.argv[3]) if bad: print("\n".join(bad)) raise SystemExit(1) -' "${CLUSTER_SCRIPT_DIR}/rewrite-namespaces.py" "${rendered}" "${OPENSHIFT_NAMESPACE}-")"; then +' "${CLUSTER_SCRIPT_DIR}/rewrite-namespaces.py" "${rendered}" "${prefix}")"; then error "Refusing cluster-scoped names that would collide with stage:" error "${bad}" exit 1 fi } -ensure_e2e_cluster_role() { - # One shared ClusterRole for every e2e environment. oc apply creates or - # patches when the rules differ. If this user cannot apply it, reuse it - # when it already exists; fail if it is missing. - local rendered="$1" - local err - assert_expected_cluster_scoped "${rendered}" - if err="$(oc_cli apply -f "${rendered}" 2>&1)"; then - info "ClusterRole hypershell-e2e applied" - printf '%s\n' "${err}" - rm -f "${rendered}" - return 0 - fi - rm -f "${rendered}" - if grep -qi 'forbidden' <<<"${err}" && oc_cli get clusterrole hypershell-e2e >/dev/null 2>&1; then - warn "Cannot update ClusterRole hypershell-e2e (forbidden); using the existing role." - return 0 - fi - fail_required_cluster_rbac "${err}" -} - apply_sandbox_scc() { local err if err="$(oc_cli apply -n "${OPENSHIFT_NAMESPACE}" -f - 2>&1 <&1)"; then - rm -f "${bindings}" + assert_expected_cluster_scoped "${rendered}" + if ! err="$(oc_cli apply -f "${rendered}" 2>&1)"; then + rm -f "${rendered}" fail_required_cluster_rbac "${err}" fi printf '%s\n' "${err}" - rm -f "${bindings}" - + rm -f "${rendered}" apply_sandbox_scc } @@ -1098,9 +1068,12 @@ cluster_down() { return 0 fi - info "Deleting this environment's ClusterRoleBindings..." - oc_cli delete clusterrolebinding "${OPENSHIFT_NAMESPACE}-hypershell-controller-scc-bind" --ignore-not-found >/dev/null 2>&1 || true - oc_cli delete clusterrolebinding "${OPENSHIFT_NAMESPACE}-hypershell-controller" --ignore-not-found >/dev/null 2>&1 || true + info "Deleting this environment's cluster-scoped RBAC..." + local prefix="${OPENSHIFT_NAMESPACE}-dev-" + oc_cli delete clusterrolebinding "${prefix}hypershell-controller-scc-bind" --ignore-not-found >/dev/null 2>&1 || true + oc_cli delete clusterrolebinding "${prefix}hypershell-controller" --ignore-not-found >/dev/null 2>&1 || true + oc_cli delete clusterrole "${prefix}hypershell-controller-scc-bind" --ignore-not-found >/dev/null 2>&1 || true + oc_cli delete clusterrole "${prefix}hypershell-controller" --ignore-not-found >/dev/null 2>&1 || true info "Removing namespace group ${OPENSHIFT_NAMESPACE} and ${OPENSHIFT_KEYCLOAK_NAMESPACE}" remove_project "${OPENSHIFT_KEYCLOAK_NAMESPACE}" diff --git a/scripts/cluster/lib_test.sh b/scripts/cluster/lib_test.sh index a53d6764..b269063a 100755 --- a/scripts/cluster/lib_test.sh +++ b/scripts/cluster/lib_test.sh @@ -79,17 +79,17 @@ else FAIL=$((FAIL + 1)) echo 'FAIL: OpenShift cluster_down does not remove the Keycloak namespace' fi -if grep -A50 '^cluster_down()' "${SCRIPT_DIR}/drivers/openshift.sh" | grep -q 'delete clusterrole '; then - FAIL=$((FAIL + 1)) - echo 'FAIL: OpenShift cluster_down deletes a ClusterRole (must not remove shared hypershell-e2e or stage)' -else +if grep -A50 '^cluster_down()' "${SCRIPT_DIR}/drivers/openshift.sh" | grep -q 'clusterrole "${prefix}hypershell-controller"'; then PASS=$((PASS + 1)) +else + FAIL=$((FAIL + 1)) + echo 'FAIL: OpenShift cluster_down does not delete this environment'\''s prefixed ClusterRole' fi -if grep -A50 '^cluster_down()' "${SCRIPT_DIR}/drivers/openshift.sh" | grep -q 'clusterrolebinding "${OPENSHIFT_NAMESPACE}-hypershell-controller"'; then +if grep -A50 '^cluster_down()' "${SCRIPT_DIR}/drivers/openshift.sh" | grep -q 'OPENSHIFT_NAMESPACE}-dev-'; then PASS=$((PASS + 1)) else FAIL=$((FAIL + 1)) - echo 'FAIL: OpenShift cluster_down does not delete this environment'\''s prefixed ClusterRoleBinding' + echo 'FAIL: OpenShift cluster_down does not use the -dev- cluster-scoped prefix' fi if grep -E 'delete clusterrole(binding)? "hypershell-controller' "${SCRIPT_DIR}/drivers/openshift.sh"; then FAIL=$((FAIL + 1)) @@ -264,11 +264,11 @@ else FAIL=$((FAIL + 1)) echo 'FAIL: keycloak-service name was rewritten' fi -if printf '%s' "${rewritten}" | grep -q 'name: alice-hypershell-controller$'; then +if printf '%s' "${rewritten}" | grep -q 'name: alice-dev-hypershell-controller$'; then PASS=$((PASS + 1)) else FAIL=$((FAIL + 1)) - echo 'FAIL: ClusterRoleBinding name not prefixed' + echo 'FAIL: ClusterRoleBinding name not prefixed with alice-dev-' fi if printf '%s' "${rewritten}" | grep -q 'controllerroleRef'; then FAIL=$((FAIL + 1)) @@ -276,19 +276,19 @@ if printf '%s' "${rewritten}" | grep -q 'controllerroleRef'; then else PASS=$((PASS + 1)) fi -# Shared ClusterRole hypershell-e2e; ClusterRoleBinding roleRef points at it +# Per-environment ClusterRole/ClusterRoleBinding: ${ns}-dev-${name} if printf '%s' "${rewritten}" | grep -q 'kind: ClusterRole' \ - && printf '%s' "${rewritten}" | awk '/kind: ClusterRole$/{p=1} p&&/^ name:/{print; exit}' | grep -q 'name: hypershell-e2e'; then + && printf '%s' "${rewritten}" | awk '/kind: ClusterRole$/{p=1} p&&/^ name:/{print; exit}' | grep -q 'name: alice-dev-hypershell-controller'; then PASS=$((PASS + 1)) else FAIL=$((FAIL + 1)) - echo 'FAIL: ClusterRole name should be hypershell-e2e' + echo 'FAIL: ClusterRole name should be alice-dev-hypershell-controller' fi -if printf '%s' "${rewritten}" | grep -A4 'roleRef:' | grep -q 'name: hypershell-e2e'; then +if printf '%s' "${rewritten}" | grep -A4 'roleRef:' | grep -q 'name: alice-dev-hypershell-controller'; then PASS=$((PASS + 1)) else FAIL=$((FAIL + 1)) - echo 'FAIL: ClusterRoleBinding roleRef.name should be hypershell-e2e' + echo 'FAIL: ClusterRoleBinding roleRef.name should be alice-dev-hypershell-controller' fi if printf '%s' "${rewritten}" | awk '/kind: ClusterRole$/{p=1} p&&/^ name:/{print; exit}' | grep -qx ' name: hypershell-controller'; then FAIL=$((FAIL + 1)) @@ -395,7 +395,7 @@ crb_out="$(python3 "${SCRIPT_DIR}/rewrite-namespaces.py" \ --only-namespace __cluster__ \ --include-cluster-scoped < "${crb_fixture}")" rm -f "${crb_fixture}" -if printf '%s' "${crb_out}" | grep -qx ' name: alice-hypershell-controller'; then +if printf '%s' "${crb_out}" | grep -qx ' name: alice-dev-hypershell-controller'; then PASS=$((PASS + 1)) else FAIL=$((FAIL + 1)) @@ -408,11 +408,11 @@ else FAIL=$((FAIL + 1)) echo 'FAIL: roleRef was not kept as its own line' fi -if printf '%s' "${crb_out}" | grep -A3 '^roleRef:' | grep -qx ' name: hypershell-e2e'; then +if printf '%s' "${crb_out}" | grep -A3 '^roleRef:' | grep -qx ' name: alice-dev-hypershell-controller'; then PASS=$((PASS + 1)) else FAIL=$((FAIL + 1)) - echo 'FAIL: ClusterRoleBinding roleRef.name should be hypershell-e2e' + echo 'FAIL: ClusterRoleBinding roleRef.name should be alice-dev-hypershell-controller' printf '%s\n' "${crb_out}" fi @@ -438,7 +438,7 @@ sys_out="$(python3 "${SCRIPT_DIR}/rewrite-namespaces.py" \ --only-namespace __cluster__ \ --include-cluster-scoped < "${sys_crb}")" rm -f "${sys_crb}" -if printf '%s' "${sys_out}" | grep -qx ' name: alice-hypershell-sandbox-scc'; then +if printf '%s' "${sys_out}" | grep -qx ' name: alice-dev-hypershell-sandbox-scc'; then PASS=$((PASS + 1)) else FAIL=$((FAIL + 1)) @@ -555,7 +555,7 @@ from importlib.machinery import SourceFileLoader from pathlib import Path import sys rw = SourceFileLoader("rewrite", sys.argv[1]).load_module() -bad = rw.unprefixed_cluster_scoped(Path(sys.argv[2]).read_text(), "alice-") +bad = rw.unprefixed_cluster_scoped(Path(sys.argv[2]).read_text(), "alice-dev-") if bad: print("\n".join(bad)) raise SystemExit(1) diff --git a/scripts/cluster/rewrite-namespaces.py b/scripts/cluster/rewrite-namespaces.py index e29d4054..d10ae290 100755 --- a/scripts/cluster/rewrite-namespaces.py +++ b/scripts/cluster/rewrite-namespaces.py @@ -4,11 +4,10 @@ Maps the overlay's platform namespace (hypershell-system) to OPENSHIFT_NAMESPACE and the bundled Keycloak namespace (keycloak) to ${OPENSHIFT_NAMESPACE}-keycloak. -ClusterRoles from the overlay are merged into one shared ClusterRole named -hypershell-e2e so every e2e environment reuses the same rules and does not patch -stage's hypershell-controller. ClusterRoleBindings stay prefixed per platform -namespace; their roleRef points at hypershell-e2e unless it is a built-in -system: ClusterRole. +ClusterRole and ClusterRoleBinding names (and ClusterRoleBinding roleRefs to +those ClusterRoles) are prefixed with ${platform}-dev- so each openshift-up +environment has its own copy and does not patch stage's hypershell-controller. +Built-in ClusterRoles (system:*) are not renamed. """ from __future__ import annotations @@ -18,10 +17,14 @@ PLATFORM_NS = "hypershell-system" KEYCLOAK_NS = "keycloak" -E2E_CLUSTER_ROLE = "hypershell-e2e" +DEV_SCOPE = "dev" CLUSTER_SCOPED_KINDS = frozenset({"ClusterRole", "ClusterRoleBinding"}) +def cluster_scoped_prefix(platform_ns: str) -> str: + return f"{platform_ns}-{DEV_SCOPE}-" + + def split_docs(text: str) -> list[str]: if text.startswith("---"): body = text[3:] @@ -58,8 +61,8 @@ def repl(match: re.Match[str]) -> str: return head + rest -def set_e2e_cluster_role_ref(doc: str) -> str: - """Point roleRef at hypershell-e2e unless it is a built-in system: ClusterRole.""" +def prefix_role_ref(doc: str, prefix: str) -> str: + """Prefix roleRef.name unless it is a built-in system: ClusterRole.""" parts = re.split(r"^(roleRef:)", doc, maxsplit=1, flags=re.M) if len(parts) < 3: return doc @@ -67,9 +70,9 @@ def set_e2e_cluster_role_ref(doc: str) -> str: def repl(match: re.Match[str]) -> str: name = match.group(2) - if name.startswith("system:") or name == E2E_CLUSTER_ROLE: + if name.startswith("system:") or name.startswith(prefix): return match.group(0) - return f"{match.group(1)}{E2E_CLUSTER_ROLE}" + return f"{match.group(1)}{prefix}{name}" rest = re.sub(r"^( name:\s*)(\S+)[^\S\n]*$", repl, rest, count=1, flags=re.M) return head + marker + rest @@ -91,9 +94,11 @@ def rewrite_doc(doc: str, platform_ns: str, keycloak_ns: str) -> str: rewritten, flags=re.M, ) + prefix = cluster_scoped_prefix(platform_ns) + if kind in CLUSTER_SCOPED_KINDS: + rewritten = prefix_metadata_name(rewritten, prefix) if kind == "ClusterRoleBinding": - rewritten = prefix_metadata_name(rewritten, f"{platform_ns}-") - rewritten = set_e2e_cluster_role_ref(rewritten) + rewritten = prefix_role_ref(rewritten, prefix) return rewritten @@ -115,61 +120,18 @@ def metadata_name(doc: str) -> str | None: def unprefixed_cluster_scoped(text: str, prefix: str) -> list[str]: - """Return cluster-scoped ids that would collide with stage or another instance. - - ClusterRole must be the shared hypershell-e2e name. ClusterRoleBinding names - must start with the platform-namespace prefix. - """ + """Return ClusterRole/ClusterRoleBinding ids whose metadata.name lacks prefix.""" bad: list[str] = [] for doc in split_docs(text): kind = kind_of(doc) + if kind not in CLUSTER_SCOPED_KINDS: + continue name = metadata_name(doc) or "" - if kind == "ClusterRole": - if name != E2E_CLUSTER_ROLE: - bad.append(f"{kind}/{name or '(missing name)'}") - elif kind == "ClusterRoleBinding": - if not name.startswith(prefix): - bad.append(f"{kind}/{name or '(missing name)'}") + if not name.startswith(prefix): + bad.append(f"{kind}/{name or '(missing name)'}") return bad -def cluster_role_rules_block(doc: str) -> str: - match = re.search(r"^rules:\n(.*)\Z", doc, re.S | re.M) - if not match: - return "" - return match.group(1).rstrip("\n") - - -def merge_e2e_cluster_role(docs: list[str]) -> list[str]: - """Collapse overlay ClusterRoles into one shared hypershell-e2e ClusterRole.""" - others: list[str] = [] - rule_blocks: list[str] = [] - for doc in docs: - if kind_of(doc) != "ClusterRole": - others.append(doc) - continue - rules = cluster_role_rules_block(doc) - if rules: - rule_blocks.append(rules) - if not rule_blocks and not any(kind_of(d) == "ClusterRole" for d in docs): - return others - rules_yaml = "\n".join(rule_blocks) if rule_blocks else " []" - merged = ( - "apiVersion: rbac.authorization.k8s.io/v1\n" - "kind: ClusterRole\n" - "metadata:\n" - f" name: {E2E_CLUSTER_ROLE}\n" - " labels:\n" - " app.kubernetes.io/name: hypershell\n" - " app.kubernetes.io/component: e2e\n" - " app.kubernetes.io/part-of: hypershell\n" - ' hypershell.redhat.io/shared-e2e: "true"\n' - "rules:\n" - f"{rules_yaml}\n" - ) - return [merged] + others - - def keep_doc( doc: str, *, @@ -213,7 +175,6 @@ def rewrite( only_kinds: set[str] | None = None, ) -> str: docs = [rewrite_doc(doc, platform_ns, keycloak_ns) for doc in split_docs(text)] - docs = merge_e2e_cluster_role(docs) docs = [ doc for doc in docs diff --git a/skills/RECONCILE.md b/skills/RECONCILE.md index e2a62e07..ec36c6de 100644 --- a/skills/RECONCILE.md +++ b/skills/RECONCILE.md @@ -136,7 +136,7 @@ Layer 7: web-console/architecture (depends on data-model, security, UI | OS-10 | Blessed OpenShift Overlay | Partial | Namespace parameterization, Routes, SCC RoleBindings; gateway base domain discovered from the shared Gateway listener (not `GATEWAY_API_BASE_DOMAIN`). Drift-check CI job deferred. | `deploy/openshift/`, `rewrite-namespaces.py` | OS-W2 | | OS-11 | OpenShift CI Workflow Shape | Missing | Intentionally deferred: not local-dev lifecycle | - | Future | | OS-12 | Cluster Infrastructure Prerequisites | Present | `make openshift-up` fails fast when the shared Gateway is missing or not Programmed. GatewayClass is cluster-scoped and not GET-checked (developers typically cannot read it). | `drivers/openshift.sh` `check_infrastructure` | OS-W2 | -| OS-13 | Cluster-Scoped Permissions + SCC/RBAC posture | Present | Shared ClusterRole `hypershell-e2e` (`oc apply` create-or-patch); per-namespace prefixed ClusterRoleBindings; fail if the role is missing and cannot be created. Overlay keeps `RBAC_ENFORCE=true`. | `rewrite-namespaces.py`, `drivers/openshift.sh` `ensure_e2e_cluster_role` | OS-W2 | +| OS-13 | Cluster-Scoped Permissions + SCC/RBAC posture | Present | Per-environment ClusterRole/ClusterRoleBinding prefixed `${ns}-dev-`; apply is required (fail if forbidden). Overlay keeps `RBAC_ENFORCE=true`. | `rewrite-namespaces.py`, `drivers/openshift.sh` `apply_required_cluster_rbac` | OS-W2 | Local-dev lifecycle (`make openshift-up` / `down` / component swaps) is implemented. E2E driver completion beyond the OS-W1 manual slice, legacy `pr-test` consolidation, ephemeral CI, access handoff, overlay drift CI, and the OpenShift e2e workflow remain out of scope for this wave. diff --git a/specs/platform/openshift-development.spec.md b/specs/platform/openshift-development.spec.md index bdfad5d3..7696fd8b 100644 --- a/specs/platform/openshift-development.spec.md +++ b/specs/platform/openshift-development.spec.md @@ -334,7 +334,7 @@ the projects. The command SHALL NOT require namespace labels in order to delete. - THEN each deployment runs in its own namespace group - AND each deployment has distinct gateway hostnames - AND neither deployment changes, conflicts with, or interacts with the other -- AND each environment's ClusterRoleBindings are prefixed with that environment's platform namespace and refer to ClusterRole `hypershell-e2e` +- AND each environment's ClusterRole and ClusterRoleBinding names are prefixed with `${OPENSHIFT_NAMESPACE}-dev-` - AND neither environment patches unprefixed names such as `hypershell-controller` that another instance (for example stage) already owns #### Scenario: Namespace cleanup removes only one deployment @@ -939,19 +939,16 @@ namespace-scoped RoleBinding CANNOT grant it, and a single pre-created ClusterRoleBinding with a fixed service-account subject cannot cover the controller service account of an unknown future ephemeral namespace. -For local-dev, `make openshift-up` is that privileged actor. It SHALL apply one -shared ClusterRole named `hypershell-e2e` (create or patch when the rules differ), -per-environment ClusterRoleBindings whose names are prefixed with the platform -namespace and whose `roleRef` is `hypershell-e2e`, and the privileged SCC RoleBinding +For local-dev, `make openshift-up` is that privileged actor. It SHALL apply +ClusterRoles and ClusterRoleBindings whose names are prefixed with +`${OPENSHIFT_NAMESPACE}-dev-`, plus the privileged SCC RoleBinding `hypershell-sandbox-scc`. Gateways and sandboxes cannot run without those grants, so -the command SHALL fail if the ClusterRole is missing and cannot be created, or if the -per-environment bindings cannot be applied. If `hypershell-e2e` already exists and -the current user cannot patch it, the command SHALL reuse the existing ClusterRole. -It SHALL NOT create, patch, or delete ClusterRole `hypershell-controller` or +the command SHALL fail if the current user cannot create those objects. It SHALL NOT +create, patch, or delete unprefixed ClusterRole `hypershell-controller` or ClusterRoleBinding `hypershell-controller`; those names belong to other instances that share the cluster, such as stage. Built-in ClusterRoles whose names start with `system:` SHALL NOT be renamed. `make openshift-down` SHALL delete only this -environment's prefixed ClusterRoleBindings, not ClusterRole `hypershell-e2e`. +environment's `${OPENSHIFT_NAMESPACE}-dev-*` ClusterRoles and ClusterRoleBindings. The deployment into the ephemeral namespace SHALL NOT attempt to grant the cluster-scoped `bind` through a namespace-scoped RoleBinding, and SHALL NOT assume the @@ -960,11 +957,10 @@ namespace. #### Scenario: Ephemeral namespace has the permissions the overlay needs -- GIVEN ClusterRole `hypershell-e2e` can be applied, or already exists -- AND the developer can apply per-namespace ClusterRoleBindings and the privileged SCC RoleBinding +- GIVEN the developer can create ClusterRoles and ClusterRoleBindings prefixed with `${OPENSHIFT_NAMESPACE}-dev-` +- AND the developer can apply the privileged SCC RoleBinding in that namespace - WHEN the developer runs `make openshift-up` -- THEN the command applies ClusterRole `hypershell-e2e` (creating it or patching it when it differs) -- AND applies ClusterRoleBindings prefixed with the platform namespace whose `roleRef` is `hypershell-e2e` +- THEN the command applies `${OPENSHIFT_NAMESPACE}-dev-hypershell-controller` and `${OPENSHIFT_NAMESPACE}-dev-hypershell-controller-scc-bind` ClusterRoles and ClusterRoleBindings - AND the command does not patch ClusterRole `hypershell-controller` or ClusterRoleBinding `hypershell-controller` - AND applies RoleBinding `hypershell-sandbox-scc` for the sandbox service account - AND the controller can create the per-namespace privileged RoleBinding for a sandbox because it was granted `bind` From b32a04207745c9500614e9dc0209726b2a35f141 Mon Sep 17 00:00:00 2001 From: Kyle Squizzato Date: Tue, 1 Sep 2026 08:02:30 -0700 Subject: [PATCH 04/14] fix(e2e-openshift): drop Fleet from seed and token-scope namespace rewrite POST /fleets 404s on current main, so seed the same top-level resources kind-up does. Rewrite hypershell-system only as a DNS label so hyphenated image names stay intact while in-cluster DNS still maps. Assisted-by: Cursor Grok 4.6 Signed-off-by: Kyle Squizzato --- scripts/cluster/drivers/openshift.sh | 54 ++++++++------------------- scripts/cluster/lib_test.sh | 28 +++++++++++++- scripts/cluster/rewrite-namespaces.py | 8 +++- skills/RECONCILE.md | 2 +- 4 files changed, 50 insertions(+), 42 deletions(-) diff --git a/scripts/cluster/drivers/openshift.sh b/scripts/cluster/drivers/openshift.sh index c6dfd48b..25868e7f 100755 --- a/scripts/cluster/drivers/openshift.sh +++ b/scripts/cluster/drivers/openshift.sh @@ -785,54 +785,30 @@ seed_via_api() { echo "${resp}" | grep -o '"id":"[^"]*"' | head -1 | cut -d'"' -f4 || true } - local seed_failed="" FLEET_ID="" CLUSTER_ID="" RELEASE_ID="" DATABASE_ID="" GATEWAY_ID="" + local seed_failed="" CLUSTER_ID="" RELEASE_ID="" DATABASE_ID="" GATEWAY_ID="" local raw http body - raw="$(api_exec GET /api/hypershell/v1/fleets)" + raw="$(api_exec GET /api/hypershell/v1/managed_clusters)" http="$(printf '%s' "${raw}" | tail -1)" body="$(printf '%s' "${raw}" | sed '$d')" if [[ "${http}" == "200" ]]; then - FLEET_ID="$(extract_named_id "${body}" default)" + CLUSTER_ID="$(extract_named_id "${body}" local-openshift)" fi - if [[ -z "${FLEET_ID}" ]]; then - info "Creating default Fleet..." - raw="$(api_exec POST /api/hypershell/v1/fleets '{"name":"default","description":"OpenShift development fleet"}')" + if [[ -z "${CLUSTER_ID}" ]]; then + info "Creating ManagedCluster..." + raw="$(api_exec POST /api/hypershell/v1/managed_clusters \ + "{\"name\":\"local-openshift\",\"provider\":\"openshift\",\"kubeconfig_secret\":\"openshift-kubeconfig\"}")" http="$(printf '%s' "${raw}" | tail -1)" body="$(printf '%s' "${raw}" | sed '$d')" - FLEET_ID="$(extract_id "${body}")" - if [[ -z "${FLEET_ID}" ]]; then - warn "Fleet creation failed (HTTP ${http}): ${body:-no response}" + CLUSTER_ID="$(extract_id "${body}")" + if [[ -z "${CLUSTER_ID}" ]]; then + warn "ManagedCluster creation failed (HTTP ${http}): ${body:-no response}" seed_failed=true else - success "Fleet created: ${FLEET_ID}" + success "ManagedCluster created: ${CLUSTER_ID}" fi else - success "default Fleet already exists: ${FLEET_ID}" - fi - - if [[ -z "${seed_failed}" ]]; then - raw="$(api_exec GET /api/hypershell/v1/managed_clusters)" - http="$(printf '%s' "${raw}" | tail -1)" - body="$(printf '%s' "${raw}" | sed '$d')" - if [[ "${http}" == "200" ]]; then - CLUSTER_ID="$(extract_named_id "${body}" local-openshift)" - fi - if [[ -z "${CLUSTER_ID}" ]]; then - info "Creating ManagedCluster..." - raw="$(api_exec POST /api/hypershell/v1/managed_clusters \ - "{\"name\":\"local-openshift\",\"fleet_id\":\"${FLEET_ID}\",\"provider\":\"openshift\",\"kubeconfig_secret\":\"openshift-kubeconfig\"}")" - http="$(printf '%s' "${raw}" | tail -1)" - body="$(printf '%s' "${raw}" | sed '$d')" - CLUSTER_ID="$(extract_id "${body}")" - if [[ -z "${CLUSTER_ID}" ]]; then - warn "ManagedCluster creation failed (HTTP ${http}): ${body:-no response}" - seed_failed=true - else - success "ManagedCluster created: ${CLUSTER_ID}" - fi - else - success "local-openshift ManagedCluster already exists: ${CLUSTER_ID}" - fi + success "local-openshift ManagedCluster already exists: ${CLUSTER_ID}" fi if [[ -z "${seed_failed}" ]]; then @@ -845,7 +821,7 @@ seed_via_api() { if [[ -z "${RELEASE_ID}" ]]; then info "Creating GatewayRelease..." raw="$(api_exec POST /api/hypershell/v1/gateway_releases \ - "{\"name\":\"dev-release\",\"fleet_id\":\"${FLEET_ID}\",\"image\":\"${GATEWAY_IMAGE}\"}")" + "{\"name\":\"dev-release\",\"image\":\"${GATEWAY_IMAGE}\"}")" http="$(printf '%s' "${raw}" | tail -1)" body="$(printf '%s' "${raw}" | sed '$d')" RELEASE_ID="$(extract_id "${body}")" @@ -870,7 +846,7 @@ seed_via_api() { if [[ -z "${DATABASE_ID}" ]]; then info "Creating ManagedDatabase..." raw="$(api_exec POST /api/hypershell/v1/managed_databases \ - "{\"name\":\"openshell-db\",\"fleet_id\":\"${FLEET_ID}\",\"provider\":\"cnpg\"}")" + "{\"name\":\"openshell-db\",\"provider\":\"cnpg\"}")" http="$(printf '%s' "${raw}" | tail -1)" body="$(printf '%s' "${raw}" | sed '$d')" if [[ "${http}" != "201" && "${http}" != "200" ]]; then @@ -897,7 +873,7 @@ seed_via_api() { local oidc oidc="{\\\"issuer\\\":\\\"${OPENSHIFT_OIDC_ISSUER}\\\",\\\"audience\\\":\\\"hypershell-frontend\\\",\\\"roles_claim\\\":\\\"groups\\\",\\\"admin_role\\\":\\\"hypershell-admins\\\",\\\"user_role\\\":\\\"hypershell-users\\\"}" raw="$(api_exec POST /api/hypershell/v1/gateways \ - "{\"name\":\"dev-gateway\",\"fleet_id\":\"${FLEET_ID}\",\"cluster_id\":\"${CLUSTER_ID}\",\"release_id\":\"${RELEASE_ID}\",\"database_id\":\"${DATABASE_ID}\",\"oidc\":\"${oidc}\"}")" + "{\"name\":\"dev-gateway\",\"cluster_id\":\"${CLUSTER_ID}\",\"release_id\":\"${RELEASE_ID}\",\"database_id\":\"${DATABASE_ID}\",\"oidc\":\"${oidc}\",\"route\":\"{\\\"enabled\\\":true}\"}")" http="$(printf '%s' "${raw}" | tail -1)" body="$(printf '%s' "${raw}" | sed '$d')" GATEWAY_ID="$(extract_id "${body}")" diff --git a/scripts/cluster/lib_test.sh b/scripts/cluster/lib_test.sh index b269063a..146dc0e1 100755 --- a/scripts/cluster/lib_test.sh +++ b/scripts/cluster/lib_test.sh @@ -230,6 +230,15 @@ subjects: - kind: ServiceAccount name: hypershell-controller namespace: hypershell-system +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: sidecar-ref + namespace: hypershell-system +data: + image: quay.io/example/hypershell-system-sidecar:1 + dns: hypershell-api-server.hypershell-system.svc.cluster.local EOF rewritten="$(python3 "${SCRIPT_DIR}/rewrite-namespaces.py" \ @@ -237,7 +246,24 @@ rewritten="$(python3 "${SCRIPT_DIR}/rewrite-namespaces.py" \ --keycloak-namespace alice-keycloak < "${fixture}")" rm -f "${fixture}" -assert_eq 0 "$(printf '%s' "${rewritten}" | grep -c hypershell-system || true)" "no leftover hypershell-system" +if printf '%s' "${rewritten}" | grep -qE '(^|[^A-Za-z0-9-])hypershell-system([^A-Za-z0-9-]|$)'; then + FAIL=$((FAIL + 1)) + echo 'FAIL: leftover hypershell-system token after rewrite' +else + PASS=$((PASS + 1)) +fi +if printf '%s' "${rewritten}" | grep -q 'quay.io/example/hypershell-system-sidecar:1'; then + PASS=$((PASS + 1)) +else + FAIL=$((FAIL + 1)) + echo 'FAIL: hyphenated image name containing hypershell-system was rewritten' +fi +if printf '%s' "${rewritten}" | grep -q 'hypershell-api-server.alice.svc.cluster.local'; then + PASS=$((PASS + 1)) +else + FAIL=$((FAIL + 1)) + echo 'FAIL: in-cluster DNS hypershell-system was not rewritten' +fi printf '%s' "${rewritten}" | grep -q $'kind: Namespace\nmetadata:\n name: alice$' \ || printf '%s' "${rewritten}" | grep -A2 'kind: Namespace' | grep -q 'name: alice' if printf '%s' "${rewritten}" | grep -q 'name: alice$'; then diff --git a/scripts/cluster/rewrite-namespaces.py b/scripts/cluster/rewrite-namespaces.py index d10ae290..945d65d0 100755 --- a/scripts/cluster/rewrite-namespaces.py +++ b/scripts/cluster/rewrite-namespaces.py @@ -19,6 +19,12 @@ KEYCLOAK_NS = "keycloak" DEV_SCOPE = "dev" CLUSTER_SCOPED_KINDS = frozenset({"ClusterRole", "ClusterRoleBinding"}) +# Rewrite the platform namespace as a DNS label, not a substring of a longer +# name (image repos, args). Still matches namespace fields, label selectors, +# env values, and in-cluster DNS like hypershell-api-server.hypershell-system.svc. +PLATFORM_NS_TOKEN = re.compile( + r"(? str: @@ -80,7 +86,7 @@ def repl(match: re.Match[str]) -> str: def rewrite_doc(doc: str, platform_ns: str, keycloak_ns: str) -> str: kind = kind_of(doc) - rewritten = doc.replace(PLATFORM_NS, platform_ns) + rewritten = PLATFORM_NS_TOKEN.sub(platform_ns, doc) rewritten = re.sub( r"^(\s*namespace:\s*)" + re.escape(KEYCLOAK_NS) + r"\s*$", rf"\g<1>{keycloak_ns}", diff --git a/skills/RECONCILE.md b/skills/RECONCILE.md index ec36c6de..7b1ef44c 100644 --- a/skills/RECONCILE.md +++ b/skills/RECONCILE.md @@ -607,7 +607,7 @@ Implemented `tests/e2e/drivers/openshift.sh` for an already-deployed environment ### Wave OS-W2: OpenShift local-dev lifecycle (up/down/swap) ✅ -Implemented `scripts/cluster/` with a driver model. `make kind-*` wraps today's `scripts/kind/` with no behavior change. `make openshift-up` deploys `kustomize build deploy/openshift/` into an ephemeral namespace group (`OPENSHIFT_NAMESPACE` + `${OPENSHIFT_NAMESPACE}-keycloak`), stamps ownership labels, refuses foreign namespaces, fails fast when the shared Gateway is missing, reads the gateway base domain from that Gateway's listener hostname, seeds Fleet/ManagedCluster/GatewayRelease/ManagedDatabase/Gateway, and prints Routes. `make openshift-down` deletes only owned namespaces. Component swaps (`make openshift-api-server-up` and siblings) build, push an immutable commit+namespace identity to the internal registry, record per-namespace state in `.openshift-swaps/`, and are preserved across reconcile. Overlay: web-console + Keycloak Routes; SCC *use* is namespace-scoped RoleBindings. E2E, CI, `pr-test` consolidation, and overlay drift CI were explicitly out of scope. +Implemented `scripts/cluster/` with a driver model. `make kind-*` wraps today's `scripts/kind/` with no behavior change. `make openshift-up` deploys `kustomize build deploy/openshift/` into an ephemeral namespace group (`OPENSHIFT_NAMESPACE` + `${OPENSHIFT_NAMESPACE}-keycloak`), stamps ownership labels, refuses foreign namespaces, fails fast when the shared Gateway is missing, reads the gateway base domain from that Gateway's listener hostname, seeds ManagedCluster/GatewayRelease/ManagedDatabase/Gateway, and prints Routes. `make openshift-down` deletes only owned namespaces. Component swaps (`make openshift-api-server-up` and siblings) build, push an immutable commit+namespace identity to the internal registry, record per-namespace state in `.openshift-swaps/`, and are preserved across reconcile. Overlay: web-console + Keycloak Routes; SCC *use* is namespace-scoped RoleBindings. E2E, CI, `pr-test` consolidation, and overlay drift CI were explicitly out of scope. ### Wave R1-R8: RBAC COMPLETED From ec5a30cdc5a8cb85d2e69d38906fc812fc6abb6b Mon Sep 17 00:00:00 2001 From: Kyle Squizzato Date: Tue, 1 Sep 2026 15:30:39 -0700 Subject: [PATCH 05/14] feat(e2e-openshift): Support using existing ClusterRole if RBAC prevents creation Assisted-by: Cursor Grok 4.6 Signed-off-by: Kyle Squizzato --- DEVELOPMENT.md | 19 +++-- Makefile | 3 +- deploy/base/controller-rbac.yaml | 2 +- deploy/openshift/scc.yaml | 37 +-------- scripts/cluster/drivers/openshift.sh | 81 ++++++++++++++++---- scripts/cluster/lib_test.sh | 60 ++++++++++++--- scripts/cluster/rewrite-namespaces.py | 25 +++++- skills/RECONCILE.md | 2 +- specs/platform/openshift-development.spec.md | 58 ++++++++++---- 9 files changed, 199 insertions(+), 88 deletions(-) diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md index ca1aad37..261925bc 100644 --- a/DEVELOPMENT.md +++ b/DEVELOPMENT.md @@ -333,19 +333,23 @@ Service. This renders `kustomize build deploy/openshift/`, maps `hypershell-system` to that platform namespace and `keycloak` to `${platform}-keycloak`, applies the -required cluster-scoped RBAC (ClusterRoles and ClusterRoleBindings named -`${namespace}-dev-*`, plus the privileged SCC RoleBinding), applies the +overlay ClusterRole and ClusterRoleBinding with names prefixed +`${namespace}-dev-*` (so an environment does not patch stage's +`hypershell-controller`), plus the privileged SCC RoleBinding, applies the manifests (with prune scoped to this environment), registers the web-console Route as the Keycloak `hypershell-frontend` redirect URI, seeds a ManagedCluster, GatewayRelease, ManagedDatabase, and Gateway from this machine against the API and Keycloak Routes (the API server image has no `curl`), and prints the API, web-console, and Keycloak Routes. The gateway base domain is read from the shared Gateway's listener hostname, not from -`GATEWAY_API_BASE_DOMAIN`. Cluster-scoped names use the `${namespace}-dev-` -prefix so an environment does not patch stage's `hypershell-controller`. Applying -that RBAC is required: `make openshift-up` fails if the current user cannot create -those ClusterRoles and ClusterRoleBindings. `make openshift-down` deletes this -environment's `${namespace}-dev-*` ClusterRoles and ClusterRoleBindings. +`GATEWAY_API_BASE_DOMAIN`. When ClusterRole create is forbidden, the command +warns and continues. `OPENSHIFT_USE_EXISTING_CLUSTERROLE=true` skips creating a +per-environment ClusterRole and instead creates `${namespace}-dev-hypershell-controller` +ClusterRoleBinding whose `roleRef` is the existing cluster-wide +`hypershell-controller` ClusterRole. It never applies unprefixed +`hypershell-controller`. `make openshift-down` deletes this environment's +`${namespace}-dev-*` ClusterRoles and ClusterRoleBindings and does not delete +stage's `hypershell-controller`. `make openshift-down` and `make openshift-teardown` are the same command. There is no OpenShift cluster to destroy. Both delete the platform project @@ -392,6 +396,7 @@ active swaps. | `KIND_NO_SUDO` | (unset) | Set to `true` to skip sudo operations | | `KIND_DNS_PORT` | `5553` | Host port for CoreDNS container | | `OPENSHIFT_NAMESPACE` | `oc project -q` | Override for the platform namespace. Unset, the current oc project is used. Max 54 chars; Keycloak lands in `${name}-keycloak`. | +| `OPENSHIFT_USE_EXISTING_CLUSTERROLE` | (unset) | Set to `true` to bind this environment to the existing ClusterRole `hypershell-controller` instead of creating `${namespace}-dev-hypershell-controller`. | | `GATEWAY_API_GATEWAY_NAME` | `openshell-grpc-gateway` | Pre-existing shared Gateway name | | `GATEWAY_API_GATEWAY_NAMESPACE` | `openshift-ingress` | Namespace of the shared Gateway | | `OPENSHIFT_IMAGE_REGISTRY` | `oc registry info` | Registry used to push swapped images | diff --git a/Makefile b/Makefile index 76c67eff..b03bb080 100644 --- a/Makefile +++ b/Makefile @@ -97,6 +97,7 @@ help: @echo " Kind uses KIND_NAMESPACE (default hypershell-system); kind-up creates the cluster." @echo " OpenShift uses the current oc project (oc project -q); OPENSHIFT_NAMESPACE overrides." @echo " OpenShift requires an existing cluster — openshift-up does not create one." + @echo " OPENSHIFT_USE_EXISTING_CLUSTERROLE=true binds to ClusterRole hypershell-controller instead of creating a per-env copy." @echo "" @echo " -up Deploy the stack (Kind also creates the cluster)" @echo " -down Remove the environment namespace(s)" @@ -318,7 +319,7 @@ export api_server_local control_plane_local web_console_local export build_version build_time export API_HOSTNAME CONSOLE_HOSTNAME HEALTH_HOSTNAME KEYCLOAK_HOSTNAME METRICS_HOSTNAME KEYCLOAK_OIDC_ISSUER export KIND_DNS_PORT -export OPENSHIFT_NAMESPACE OPENSHIFT_IMAGE_REGISTRY +export OPENSHIFT_NAMESPACE OPENSHIFT_IMAGE_REGISTRY OPENSHIFT_USE_EXISTING_CLUSTERROLE export GATEWAY_API_GATEWAY_NAME GATEWAY_API_GATEWAY_NAMESPACE GATEWAY_IMAGE # Build cloud-provider-kind from a fork that adds BackendTLSPolicy support diff --git a/deploy/base/controller-rbac.yaml b/deploy/base/controller-rbac.yaml index d37e431b..698a9ae6 100644 --- a/deploy/base/controller-rbac.yaml +++ b/deploy/base/controller-rbac.yaml @@ -17,7 +17,7 @@ rules: verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] - apiGroups: ["rbac.authorization.k8s.io"] resources: ["roles", "rolebindings", "clusterroles", "clusterrolebindings"] - verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] + verbs: ["bind", "get", "list", "watch", "create", "update", "patch", "delete"] - apiGroups: ["networking.k8s.io"] resources: ["networkpolicies"] verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] diff --git a/deploy/openshift/scc.yaml b/deploy/openshift/scc.yaml index 25d66b6e..fb414c98 100644 --- a/deploy/openshift/scc.yaml +++ b/deploy/openshift/scc.yaml @@ -17,8 +17,10 @@ subjects: --- # Sandbox pods require the privileged SCC for container-in-container # execution (see specs/platform/openshell-gateway.spec.md §OpenShift). -# The controller creates additional per-tenant RoleBindings at runtime -# (reconcileOpenShiftSCC) once a privileged actor has granted it bind. +# This RoleBinding covers the sandbox SA in the controller namespace. Each +# gateway namespace gets a privileged RoleBinding at runtime +# (reconcileOpenShiftSCC). That requires bind on clusterroles, which the +# controller ClusterRole grants (same shape as stage). apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: @@ -32,34 +34,3 @@ subjects: - kind: ServiceAccount name: openshell-gateway-sandbox namespace: hypershell-system ---- -# The static binding above only covers the sandbox SA in the controller's own -# namespace. In practice each gateway lands in its own namespace, so the -# controller creates a per-namespace RoleBinding to the privileged SCC at -# runtime (reconcileOpenShiftSCC). Kubernetes escalation prevention only lets it -# create that binding if it can "bind" the referenced ClusterRole -- grant the -# controller `bind` scoped to exactly the privileged SCC ClusterRole. Using -# `bind` (rather than binding the controller to privileged directly) lets it -# grant the SCC without its own pods being able to use it. -apiVersion: rbac.authorization.k8s.io/v1 -kind: ClusterRole -metadata: - name: hypershell-controller-scc-bind -rules: - - apiGroups: ["rbac.authorization.k8s.io"] - resources: ["clusterroles"] - resourceNames: ["system:openshift:scc:privileged"] - verbs: ["bind"] ---- -apiVersion: rbac.authorization.k8s.io/v1 -kind: ClusterRoleBinding -metadata: - name: hypershell-controller-scc-bind -roleRef: - apiGroup: rbac.authorization.k8s.io - kind: ClusterRole - name: hypershell-controller-scc-bind -subjects: - - kind: ServiceAccount - name: hypershell-controller - namespace: hypershell-system diff --git a/scripts/cluster/drivers/openshift.sh b/scripts/cluster/drivers/openshift.sh index 25868e7f..742b3397 100755 --- a/scripts/cluster/drivers/openshift.sh +++ b/scripts/cluster/drivers/openshift.sh @@ -266,13 +266,26 @@ check_infrastructure() { info "Gateway base domain: ${GATEWAY_API_BASE_DOMAIN} (from ${gw_ns}/${gw_name} listener)" } -fail_required_cluster_rbac() { +use_existing_clusterrole() { + case "${OPENSHIFT_USE_EXISTING_CLUSTERROLE:-}" in + true|TRUE|1|yes|YES) return 0 ;; + *) return 1 ;; + esac +} + +warn_skipped_cluster_rbac() { local err="$1" - error "Cluster-scoped RBAC is required to provision gateways and sandboxes." - error "${err}" - error "The current user must be able to create ClusterRole and ClusterRoleBinding (${OPENSHIFT_NAMESPACE}-dev-*)." - error "Do not apply unprefixed hypershell-controller; that belongs to stage." - exit 1 + OPENSHIFT_CLUSTER_RBAC_APPLIED=false + warn "Could not apply cluster-scoped RBAC for this environment." + warn "${err}" + if use_existing_clusterrole; then + warn "OPENSHIFT_USE_EXISTING_CLUSTERROLE requires ClusterRole hypershell-controller and permission to create a prefixed ClusterRoleBinding to it." + else + warn "Expected this environment's ClusterRole and ClusterRoleBinding:" + warn " ${OPENSHIFT_NAMESPACE}-dev-hypershell-controller" + fi + warn "Gateways and sandboxes will not provision until this environment's controller is bound." + warn "Do not apply unprefixed ClusterRoleBinding hypershell-controller; that belongs to stage." } assert_expected_cluster_scoped() { @@ -321,21 +334,56 @@ EOF printf '%s\n' "${err}" return 0 fi - fail_required_cluster_rbac "${err}" + if grep -qi 'forbidden' <<<"${err}"; then + warn "Skipping privileged SCC RoleBinding (the current user cannot bind system:openshift:scc:privileged)." + warn "Gateway sandboxes will not start until an administrator creates hypershell-sandbox-scc in ${OPENSHIFT_NAMESPACE}." + return 0 + fi + error "${err}" + return 1 } -apply_required_cluster_rbac() { - info "Applying required cluster-scoped RBAC..." +apply_cluster_rbac() { + # Default: apply the overlay ClusterRole and ClusterRoleBinding, names prefixed + # so they never replace stage's hypershell-controller. Workaround: bind this + # environment's service account to the existing cluster-wide ClusterRole. + local prefix="${OPENSHIFT_NAMESPACE}-dev-" + local render_args=( + --only-namespace __cluster__ + --include-cluster-scoped + ) local rendered err - if ! rendered="$(render_openshift_manifests \ - --only-namespace __cluster__ \ - --include-cluster-scoped)"; then + OPENSHIFT_CLUSTER_RBAC_APPLIED=true + + if use_existing_clusterrole; then + info "OPENSHIFT_USE_EXISTING_CLUSTERROLE: looking up ClusterRole hypershell-controller" + if ! oc_cli get clusterrole hypershell-controller >/dev/null 2>&1; then + error "OPENSHIFT_USE_EXISTING_CLUSTERROLE is set but ClusterRole hypershell-controller was not found." + exit 1 + fi + info "Binding this environment to ClusterRole hypershell-controller (${prefix}hypershell-controller)" + render_args+=( + --omit-kinds ClusterRole + --keep-role-refs + --omit-names "${prefix}hypershell-controller-scc-bind" + ) + else + info "Applying cluster-scoped RBAC from deploy/openshift (${prefix}*)..." + fi + + if ! rendered="$(render_openshift_manifests "${render_args[@]}")"; then exit 1 fi assert_expected_cluster_scoped "${rendered}" if ! err="$(oc_cli apply -f "${rendered}" 2>&1)"; then rm -f "${rendered}" - fail_required_cluster_rbac "${err}" + if grep -qiE 'forbidden|not found' <<<"${err}"; then + warn_skipped_cluster_rbac "${err}" + apply_sandbox_scc || true + return 0 + fi + error "${err}" + return 1 fi printf '%s\n' "${err}" rm -f "${rendered}" @@ -906,6 +954,11 @@ print_banner() { info "API Server Logs: oc logs -f -l app=hypershell-api-server -n ${OPENSHIFT_NAMESPACE}" info "Control Plane Logs: oc logs -f -l app=hypershell-controller -n ${OPENSHIFT_NAMESPACE}" info "Web Console Logs: oc logs -f -l app=hypershell-web-console -n ${OPENSHIFT_NAMESPACE}" + if [[ "${OPENSHIFT_CLUSTER_RBAC_APPLIED:-true}" != "true" ]]; then + echo "" + warn "This environment's controller is not bound to a ClusterRole." + warn "Gateways/sandboxes need ${OPENSHIFT_NAMESPACE}-dev-hypershell-controller, or OPENSHIFT_USE_EXISTING_CLUSTERROLE=true plus ClusterRole hypershell-controller." + fi } cluster_up() { @@ -919,7 +972,7 @@ cluster_up() { check_infrastructure ensure_namespace_group create_bootstrap_secrets - apply_required_cluster_rbac + apply_cluster_rbac apply_overlay restore_swaps_after_reconcile configure_oidc_from_routes diff --git a/scripts/cluster/lib_test.sh b/scripts/cluster/lib_test.sh index 146dc0e1..432c369f 100755 --- a/scripts/cluster/lib_test.sh +++ b/scripts/cluster/lib_test.sh @@ -79,6 +79,12 @@ else FAIL=$((FAIL + 1)) echo 'FAIL: OpenShift cluster_down does not remove the Keycloak namespace' fi +if grep -A50 '^cluster_down()' "${SCRIPT_DIR}/drivers/openshift.sh" | grep -q 'clusterrolebinding "${prefix}hypershell-controller"'; then + PASS=$((PASS + 1)) +else + FAIL=$((FAIL + 1)) + echo 'FAIL: OpenShift cluster_down does not delete this environment'\''s prefixed ClusterRoleBinding' +fi if grep -A50 '^cluster_down()' "${SCRIPT_DIR}/drivers/openshift.sh" | grep -q 'clusterrole "${prefix}hypershell-controller"'; then PASS=$((PASS + 1)) else @@ -97,23 +103,25 @@ if grep -E 'delete clusterrole(binding)? "hypershell-controller' "${SCRIPT_DIR}/ else PASS=$((PASS + 1)) fi -if grep -q 'optional cluster-scoped RBAC (best-effort)' "${SCRIPT_DIR}/drivers/openshift.sh"; then - FAIL=$((FAIL + 1)) - echo 'FAIL: OpenShift still treats cluster-scoped RBAC as best-effort' -else +if grep -q 'OPENSHIFT_USE_EXISTING_CLUSTERROLE' "${SCRIPT_DIR}/drivers/openshift.sh" \ + && grep -A50 '^apply_cluster_rbac()' "${SCRIPT_DIR}/drivers/openshift.sh" | grep -q 'get clusterrole hypershell-controller' \ + && grep -A50 '^apply_cluster_rbac()' "${SCRIPT_DIR}/drivers/openshift.sh" | grep -q 'keep-role-refs'; then PASS=$((PASS + 1)) -fi -if grep -q 'Skipping ClusterRole' "${SCRIPT_DIR}/drivers/openshift.sh"; then - FAIL=$((FAIL + 1)) - echo 'FAIL: OpenShift still skips ClusterRole apply' else - PASS=$((PASS + 1)) + FAIL=$((FAIL + 1)) + echo 'FAIL: OPENSHIFT_USE_EXISTING_CLUSTERROLE does not look up ClusterRole hypershell-controller' fi -if grep -A80 '^apply_required_cluster_rbac()' "${SCRIPT_DIR}/drivers/openshift.sh" | grep -q 'fail_required_cluster_rbac'; then +if grep -A50 '^apply_cluster_rbac()' "${SCRIPT_DIR}/drivers/openshift.sh" | grep -q 'Applying cluster-scoped RBAC from deploy/openshift'; then PASS=$((PASS + 1)) else FAIL=$((FAIL + 1)) - echo 'FAIL: apply_required_cluster_rbac does not fail when cluster RBAC cannot be applied' + echo 'FAIL: default apply_cluster_rbac does not apply overlay ClusterRole/ClusterRoleBinding' +fi +if grep -q 'fail_required_cluster_rbac' "${SCRIPT_DIR}/drivers/openshift.sh"; then + FAIL=$((FAIL + 1)) + echo 'FAIL: OpenShift still fails the whole up when cluster RBAC cannot be applied' +else + PASS=$((PASS + 1)) fi assert_eq "alice-keycloak" "$(keycloak_namespace_for alice)" "keycloak namespace suffix" assert_ok "derived keycloak ns fits 63" validate_rfc1123_label "$(keycloak_namespace_for "$(printf 'a%.0s' {1..54})")" 63 @@ -441,6 +449,36 @@ else echo 'FAIL: ClusterRoleBinding roleRef.name should be alice-dev-hypershell-controller' printf '%s\n' "${crb_out}" fi +keep_ref_out="$(python3 "${SCRIPT_DIR}/rewrite-namespaces.py" \ + --platform-namespace alice \ + --keycloak-namespace alice-keycloak \ + --omit-namespaces \ + --keep-role-refs \ + --only-namespace __cluster__ \ + --include-cluster-scoped \ + --omit-kinds ClusterRole <<'EOF' +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: hypershell-controller +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: hypershell-controller +subjects: +- kind: ServiceAccount + name: hypershell-controller + namespace: hypershell-system +EOF +)" +if printf '%s' "${keep_ref_out}" | grep -qx ' name: alice-dev-hypershell-controller' \ + && printf '%s' "${keep_ref_out}" | grep -A3 '^roleRef:' | grep -qx ' name: hypershell-controller'; then + PASS=$((PASS + 1)) +else + FAIL=$((FAIL + 1)) + echo 'FAIL: --keep-role-refs should prefix ClusterRoleBinding name but leave roleRef hypershell-controller' + printf '%s\n' "${keep_ref_out}" +fi sys_crb="$(mktemp)" cat > "${sys_crb}" <<'EOF' diff --git a/scripts/cluster/rewrite-namespaces.py b/scripts/cluster/rewrite-namespaces.py index 945d65d0..1d7882f9 100755 --- a/scripts/cluster/rewrite-namespaces.py +++ b/scripts/cluster/rewrite-namespaces.py @@ -7,7 +7,9 @@ ClusterRole and ClusterRoleBinding names (and ClusterRoleBinding roleRefs to those ClusterRoles) are prefixed with ${platform}-dev- so each openshift-up environment has its own copy and does not patch stage's hypershell-controller. -Built-in ClusterRoles (system:*) are not renamed. +Built-in ClusterRoles (system:*) are not renamed. --keep-role-refs leaves +roleRef pointing at the existing cluster-wide ClusterRole (workaround for +OPENSHIFT_USE_EXISTING_CLUSTERROLE). """ from __future__ import annotations @@ -84,7 +86,12 @@ def repl(match: re.Match[str]) -> str: return head + marker + rest -def rewrite_doc(doc: str, platform_ns: str, keycloak_ns: str) -> str: +def rewrite_doc( + doc: str, + platform_ns: str, + keycloak_ns: str, + keep_role_refs: bool = False, +) -> str: kind = kind_of(doc) rewritten = PLATFORM_NS_TOKEN.sub(platform_ns, doc) rewritten = re.sub( @@ -103,7 +110,7 @@ def rewrite_doc(doc: str, platform_ns: str, keycloak_ns: str) -> str: prefix = cluster_scoped_prefix(platform_ns) if kind in CLUSTER_SCOPED_KINDS: rewritten = prefix_metadata_name(rewritten, prefix) - if kind == "ClusterRoleBinding": + if kind == "ClusterRoleBinding" and not keep_role_refs: rewritten = prefix_role_ref(rewritten, prefix) return rewritten @@ -179,8 +186,12 @@ def rewrite( omit_kinds: set[str] | None = None, omit_names: set[str] | None = None, only_kinds: set[str] | None = None, + keep_role_refs: bool = False, ) -> str: - docs = [rewrite_doc(doc, platform_ns, keycloak_ns) for doc in split_docs(text)] + docs = [ + rewrite_doc(doc, platform_ns, keycloak_ns, keep_role_refs=keep_role_refs) + for doc in split_docs(text) + ] docs = [ doc for doc in docs @@ -244,6 +255,11 @@ def main() -> int: default="", help="Comma-separated metadata.names to drop (e.g. hypershell-sandbox-scc).", ) + parser.add_argument( + "--keep-role-refs", + action="store_true", + help="Prefix ClusterRoleBinding names but leave roleRef pointing at the existing ClusterRole (hypershell-controller).", + ) parser.add_argument( "--strip-openshift-uids", action="store_true", @@ -263,6 +279,7 @@ def main() -> int: omit_kinds=omit_kinds or None, omit_names=omit_names or None, only_kinds=only_kinds or None, + keep_role_refs=args.keep_role_refs, ) if args.strip_openshift_uids: rendered = strip_openshift_fixed_uids(rendered) diff --git a/skills/RECONCILE.md b/skills/RECONCILE.md index 7b1ef44c..8249c9db 100644 --- a/skills/RECONCILE.md +++ b/skills/RECONCILE.md @@ -136,7 +136,7 @@ Layer 7: web-console/architecture (depends on data-model, security, UI | OS-10 | Blessed OpenShift Overlay | Partial | Namespace parameterization, Routes, SCC RoleBindings; gateway base domain discovered from the shared Gateway listener (not `GATEWAY_API_BASE_DOMAIN`). Drift-check CI job deferred. | `deploy/openshift/`, `rewrite-namespaces.py` | OS-W2 | | OS-11 | OpenShift CI Workflow Shape | Missing | Intentionally deferred: not local-dev lifecycle | - | Future | | OS-12 | Cluster Infrastructure Prerequisites | Present | `make openshift-up` fails fast when the shared Gateway is missing or not Programmed. GatewayClass is cluster-scoped and not GET-checked (developers typically cannot read it). | `drivers/openshift.sh` `check_infrastructure` | OS-W2 | -| OS-13 | Cluster-Scoped Permissions + SCC/RBAC posture | Present | Per-environment ClusterRole/ClusterRoleBinding prefixed `${ns}-dev-`; apply is required (fail if forbidden). Overlay keeps `RBAC_ENFORCE=true`. | `rewrite-namespaces.py`, `drivers/openshift.sh` `apply_required_cluster_rbac` | OS-W2 | +| OS-13 | Cluster-Scoped Permissions + SCC/RBAC posture | Present | Default applies prefixed overlay ClusterRole+ClusterRoleBinding (`bind` on clusterroles, same shape as stage). `OPENSHIFT_USE_EXISTING_CLUSTERROLE=true` looks up ClusterRole `hypershell-controller` and binds a prefixed CRB to it. Never touches unprefixed `hypershell-controller`. Down deletes this env's prefixed ClusterRole/CRB. | `rewrite-namespaces.py`, `drivers/openshift.sh` `apply_cluster_rbac`, `deploy/base/controller-rbac.yaml` | OS-W2 | Local-dev lifecycle (`make openshift-up` / `down` / component swaps) is implemented. E2E driver completion beyond the OS-W1 manual slice, legacy `pr-test` consolidation, ephemeral CI, access handoff, overlay drift CI, and the OpenShift e2e workflow remain out of scope for this wave. diff --git a/specs/platform/openshift-development.spec.md b/specs/platform/openshift-development.spec.md index 7696fd8b..67712def 100644 --- a/specs/platform/openshift-development.spec.md +++ b/specs/platform/openshift-development.spec.md @@ -939,16 +939,31 @@ namespace-scoped RoleBinding CANNOT grant it, and a single pre-created ClusterRoleBinding with a fixed service-account subject cannot cover the controller service account of an unknown future ephemeral namespace. -For local-dev, `make openshift-up` is that privileged actor. It SHALL apply -ClusterRoles and ClusterRoleBindings whose names are prefixed with -`${OPENSHIFT_NAMESPACE}-dev-`, plus the privileged SCC RoleBinding -`hypershell-sandbox-scc`. Gateways and sandboxes cannot run without those grants, so -the command SHALL fail if the current user cannot create those objects. It SHALL NOT -create, patch, or delete unprefixed ClusterRole `hypershell-controller` or -ClusterRoleBinding `hypershell-controller`; those names belong to other instances -that share the cluster, such as stage. Built-in ClusterRoles whose names start with -`system:` SHALL NOT be renamed. `make openshift-down` SHALL delete only this -environment's `${OPENSHIFT_NAMESPACE}-dev-*` ClusterRoles and ClusterRoleBindings. +For local-dev on a shared cluster, `make openshift-up` SHALL apply the +ClusterRole and ClusterRoleBinding from `deploy/openshift/` after prefixing +their names with `${OPENSHIFT_NAMESPACE}-dev-`. The ClusterRole SHALL include +`bind` on `clusterroles` and `clusterrolebindings`, matching the stage +controller ClusterRole. The command SHALL NOT create, patch, or delete +unprefixed ClusterRole `hypershell-controller` or ClusterRoleBinding +`hypershell-controller`; those names belong to other instances that share the +cluster, such as stage. Built-in ClusterRoles whose names start with `system:` +SHALL NOT be renamed. + +When `OPENSHIFT_USE_EXISTING_CLUSTERROLE` is `true`, the command SHALL look up +ClusterRole `hypershell-controller` and SHALL apply a ClusterRoleBinding named +`${OPENSHIFT_NAMESPACE}-dev-hypershell-controller` whose `roleRef` is that +existing ClusterRole and whose subject is this environment's controller +service account. It SHALL NOT create a ClusterRole, and it SHALL NOT look for +`hypershell-controller-scc-bind`. If ClusterRole `hypershell-controller` is +missing, the command SHALL fail. + +Kubernetes escalation prevention forbids a typical developer from granting +those ClusterRoles, so when ClusterRole or ClusterRoleBinding apply is +Forbidden, the command SHALL warn and continue with the rest of the stack. +Gateways and sandboxes will not provision until this environment's controller +is bound. `make openshift-down` SHALL delete this environment's +`${OPENSHIFT_NAMESPACE}-dev-*` ClusterRoles and ClusterRoleBindings and SHALL +NOT delete unprefixed `hypershell-controller`. The deployment into the ephemeral namespace SHALL NOT attempt to grant the cluster-scoped `bind` through a namespace-scoped RoleBinding, and SHALL NOT assume the @@ -960,19 +975,30 @@ namespace. - GIVEN the developer can create ClusterRoles and ClusterRoleBindings prefixed with `${OPENSHIFT_NAMESPACE}-dev-` - AND the developer can apply the privileged SCC RoleBinding in that namespace - WHEN the developer runs `make openshift-up` -- THEN the command applies `${OPENSHIFT_NAMESPACE}-dev-hypershell-controller` and `${OPENSHIFT_NAMESPACE}-dev-hypershell-controller-scc-bind` ClusterRoles and ClusterRoleBindings -- AND the command does not patch ClusterRole `hypershell-controller` or ClusterRoleBinding `hypershell-controller` +- THEN the command applies ClusterRole and ClusterRoleBinding `${OPENSHIFT_NAMESPACE}-dev-hypershell-controller` from `deploy/openshift/` +- AND the command does not create or patch ClusterRole `hypershell-controller` or ClusterRoleBinding `hypershell-controller` - AND applies RoleBinding `hypershell-sandbox-scc` for the sandbox service account - AND the controller can create the per-namespace privileged RoleBinding for a sandbox because it was granted `bind` - AND the deployment does not attempt to grant the cluster-scoped `bind` through a namespace-scoped RoleBinding +#### Scenario: Shared cluster ClusterRole workaround + +- GIVEN ClusterRole `hypershell-controller` already exists on the cluster +- AND `OPENSHIFT_USE_EXISTING_CLUSTERROLE=true` +- AND the developer can create ClusterRoleBindings prefixed with `${OPENSHIFT_NAMESPACE}-dev-` +- WHEN the developer runs `make openshift-up` +- THEN the command looks up ClusterRole `hypershell-controller` +- AND applies ClusterRoleBinding `${OPENSHIFT_NAMESPACE}-dev-hypershell-controller` with `roleRef` `hypershell-controller` +- AND the command does not create a ClusterRole +- AND the command does not create or patch ClusterRoleBinding `hypershell-controller` + #### Scenario: Cluster-scoped RBAC cannot be applied -- GIVEN the current user cannot create ClusterRoleBindings or grant the privileged SCC +- GIVEN the current user cannot create ClusterRoles or ClusterRoleBindings - WHEN the developer runs `make openshift-up` -- THEN the command fails -- AND the error states that cluster-scoped RBAC is required to provision gateways and sandboxes -- AND the command does not report the overlay as applied +- THEN the command warns that this environment is not bound +- AND the command continues and applies the namespaced overlay +- AND gateways and sandboxes will not provision until this environment's controller is bound ### Requirement: OpenShift Security Context and RBAC Parity From 40ca5c804f39f267b4f58fc66aa4782519b6494a Mon Sep 17 00:00:00 2001 From: Kyle Squizzato Date: Wed, 2 Sep 2026 08:05:51 -0700 Subject: [PATCH 06/14] fix(control-plane): scope gateway namespace GC to this controller Periodic GC listed every hypershell-managed namespace on the cluster, so an e2e controller treated stage gateways as orphans. Stamp hypershell.redhat.io/instance from the controller pod's own namespace and only reap namespaces that carry that identity. Assisted-by: Cursor Grok 4.6 Signed-off-by: Kyle Squizzato --- .github/workflows/e2e.yml | 6 +- DEVELOPMENT.md | 2 + Makefile | 15 +- README.md | 2 +- .../cmd/hypershell-controller/main.go | 2 +- .../internal/gateway/namespace.go | 151 ++++++++++++--- .../internal/gateway/namespace_test.go | 172 ++++++++++++++---- .../internal/gateway/reconciler.go | 38 +--- .../managed_database_lifecycle_test.go | 8 +- .../reconciler/managed_database_test.go | 4 +- .../internal/reconciler/namespace.go | 21 ++- .../internal/reconciler/namespace_test.go | 111 +++++++++++ .../internal/reconciler/reconciler.go | 71 +++----- deploy/base/controller.yaml | 4 +- scripts/cluster/drivers/kind.sh | 4 + scripts/cluster/drivers/openshift.sh | 35 +++- scripts/cluster/lib.sh | 21 ++- scripts/cluster/lib_test.sh | 14 +- scripts/cluster/seed.sh | 8 + scripts/kind/lib.sh | 15 ++ scripts/kind/seed.sh | 11 +- scripts/kind/up.sh | 6 +- specs/platform/e2e-testing.spec.md | 7 +- .../openshell-gateway-namespace-gc.spec.md | 123 +++++++++++-- specs/platform/openshift-development.spec.md | 2 +- tests/e2e/e2e-openshell.sh | 6 +- 26 files changed, 662 insertions(+), 197 deletions(-) create mode 100755 scripts/cluster/seed.sh diff --git a/.github/workflows/e2e.yml b/.github/workflows/e2e.yml index 9690043e..34ad440c 100644 --- a/.github/workflows/e2e.yml +++ b/.github/workflows/e2e.yml @@ -395,7 +395,7 @@ jobs: # contract change (e.g. a removed/added required field) against that # stale image would 400 and abort. The "Seed platform resources" step # runs it once the working-tree images are swapped in. - KIND_SKIP_SEED: "true" + SKIP_SEED: "true" run: make kind-up # Block until each triggered component build has concluded. @@ -460,12 +460,12 @@ jobs: # Seed platform resources now that the working-tree images are live (the # swap above rolled them in). Deferred from kind-up so the seed exercises # this PR's request contract instead of the baseline placeholder image. - # KIND_SEED_STRICT fails the job here with the real HTTP error if a create + # SEED_STRICT fails the job here with the real HTTP error if a create # is rejected, rather than surfacing later as a confusing discovery miss. - name: Seed platform resources env: DATABASE_PROVIDER: ${{ matrix.database-provider }} - KIND_SEED_STRICT: "true" + SEED_STRICT: "true" run: make kind-seed - name: Run e2e tests diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md index 261925bc..21c43d5f 100644 --- a/DEVELOPMENT.md +++ b/DEVELOPMENT.md @@ -397,6 +397,8 @@ active swaps. | `KIND_DNS_PORT` | `5553` | Host port for CoreDNS container | | `OPENSHIFT_NAMESPACE` | `oc project -q` | Override for the platform namespace. Unset, the current oc project is used. Max 54 chars; Keycloak lands in `${name}-keycloak`. | | `OPENSHIFT_USE_EXISTING_CLUSTERROLE` | (unset) | Set to `true` to bind this environment to the existing ClusterRole `hypershell-controller` instead of creating `${namespace}-dev-hypershell-controller`. | +| `SKIP_SEED` | (unset) | Set to `true` to skip ManagedCluster/GatewayRelease/ManagedDatabase/Gateway seeding on `make kind-up` and `make openshift-up`. `KIND_SKIP_SEED` is still accepted. | +| `SEED_STRICT` | (unset) | Set to `true` to fail `make kind-up` / `make kind-seed` / `make openshift-up` if seeding is incomplete. `KIND_SEED_STRICT` is still accepted. | | `GATEWAY_API_GATEWAY_NAME` | `openshell-grpc-gateway` | Pre-existing shared Gateway name | | `GATEWAY_API_GATEWAY_NAMESPACE` | `openshift-ingress` | Namespace of the shared Gateway | | `OPENSHIFT_IMAGE_REGISTRY` | `oc registry info` | Registry used to push swapped images | diff --git a/Makefile b/Makefile index b03bb080..b0ac7749 100644 --- a/Makefile +++ b/Makefile @@ -71,7 +71,7 @@ KIND_DNS_PORT?=5553 # The gateway base domain is discovered from that Gateway's listener hostname. GATEWAY_API_GATEWAY_NAME?=openshell-grpc-gateway GATEWAY_API_GATEWAY_NAMESPACE?=openshift-ingress -GATEWAY_IMAGE?=ghcr.io/nvidia/openshell/gateway:0.0.109 +GATEWAY_IMAGE?=quay.io/opendatahub/odh-openshell-gateway:v0.0.109-rhaiv.0@sha256:a80b79e514826e8d57ea137749cf18a6e7f3d92e26bfefe005f3a9c4a55b8bdd # Service hostnames (routed through the networking Gateway) API_HOSTNAME=api.hypershell.localhost @@ -96,7 +96,7 @@ help: @echo "" @echo " Kind uses KIND_NAMESPACE (default hypershell-system); kind-up creates the cluster." @echo " OpenShift uses the current oc project (oc project -q); OPENSHIFT_NAMESPACE overrides." - @echo " OpenShift requires an existing cluster — openshift-up does not create one." + @echo " OpenShift requires an existing cluster; openshift-up does not create one." @echo " OPENSHIFT_USE_EXISTING_CLUSTERROLE=true binds to ClusterRole hypershell-controller instead of creating a per-env copy." @echo "" @echo " -up Deploy the stack (Kind also creates the cluster)" @@ -113,7 +113,8 @@ help: @echo " Kind Specific" @echo " kind-env Print environment variables for local setup" @echo " kind-seed Seed platform resources into a running cluster" - @echo " KIND_SKIP_SEED=true: defer seeding during kind-up" + @echo " SKIP_SEED=true: defer seeding during kind-up / openshift-up" + @echo " SEED_STRICT=true: fail the command if seeding is incomplete" @echo " kind-fix-ports Re-establish host port forwarding (443 + 8080)" @echo " kind-gateway-trust Print SSL_CERT_FILE export so the openshell CLI trusts the dev CA" @echo " LOCAL_IMAGES=true Build baseline images from the working tree (kind-up)" @@ -319,7 +320,7 @@ export api_server_local control_plane_local web_console_local export build_version build_time export API_HOSTNAME CONSOLE_HOSTNAME HEALTH_HOSTNAME KEYCLOAK_HOSTNAME METRICS_HOSTNAME KEYCLOAK_OIDC_ISSUER export KIND_DNS_PORT -export OPENSHIFT_NAMESPACE OPENSHIFT_IMAGE_REGISTRY OPENSHIFT_USE_EXISTING_CLUSTERROLE +export OPENSHIFT_NAMESPACE OPENSHIFT_IMAGE_REGISTRY OPENSHIFT_USE_EXISTING_CLUSTERROLE SKIP_SEED SEED_STRICT export GATEWAY_API_GATEWAY_NAME GATEWAY_API_GATEWAY_NAMESPACE GATEWAY_IMAGE # Build cloud-provider-kind from a fork that adds BackendTLSPolicy support @@ -406,7 +407,7 @@ kind-up: .PHONY: kind-seed kind-seed: - @scripts/kind/seed.sh + @CLUSTER_DRIVER=kind scripts/cluster/seed.sh .PHONY: kind-down kind-down: @@ -460,6 +461,10 @@ kind-gateway-trust: openshift-up: @CLUSTER_DRIVER=openshift scripts/cluster/up.sh +.PHONY: openshift-seed +openshift-seed: + @CLUSTER_DRIVER=openshift scripts/cluster/seed.sh + .PHONY: openshift-down openshift-down: @CLUSTER_DRIVER=openshift scripts/cluster/down.sh diff --git a/README.md b/README.md index b0f5779f..1b242309 100644 --- a/README.md +++ b/README.md @@ -163,7 +163,7 @@ Or apply a kustomize patch via `deploy/openshift/kustomization.yaml` with your s |---|---|---|---| | `HYPERSHELL_GRPC_SERVER_ADDR` | `localhost:9000` | ✓ | gRPC address of the API server | | `HYPERSHELL_API_SERVER_URL` | `http://localhost:8000` | ✓ | HTTP address of the API server | -| `HYPERSHELL_NAMESPACE` | `hypershell-system` | ✓ | Namespace the control plane runs in (used for trusted CA bundle source) | +| `HYPERSHELL_NAMESPACE` | pod namespace | ✓ | Namespace this controller runs in. In cluster this is the downward API (`metadata.namespace`), so the value is unique to that controller. It is also the `hypershell.redhat.io/instance` identity stamped on gateway namespaces so GC never reaps another HyperShell's workloads. | | `GATEWAY_IMAGE` | *(none)* | **✓ required** | Container image for tenant gateways (pinned by digest; no fallback). Set in `deploy/base/controller.yaml` | | `GATEWAY_SUPERVISOR_IMAGE` | *(none)* | **✓ required** | Container image for gateway supervisors (pinned by digest; no fallback). Set in `deploy/base/controller.yaml` | | `GATEWAY_API_GATEWAY_NAME` | *(required)* | ✓ | Name of the pre-existing Gateway resource that tenant GRPCRoutes attach to | diff --git a/components/control-plane/cmd/hypershell-controller/main.go b/components/control-plane/cmd/hypershell-controller/main.go index 29a1f566..2046e0d0 100644 --- a/components/control-plane/cmd/hypershell-controller/main.go +++ b/components/control-plane/cmd/hypershell-controller/main.go @@ -166,7 +166,7 @@ func main() { clusterReconciler := reconciler.NewManagedClusterReconciler() var databaseReconciler watcher.Handler[*pb.ManagedDatabase] if managedDatabaseWatchEligible(clientset, dynamicClient) { - databaseReconciler = reconciler.NewManagedDatabaseReconciler(dynamicClient, clientset, conn) + databaseReconciler = reconciler.NewManagedDatabaseReconciler(dynamicClient, clientset, conn, cfg.Namespace) } else { log.Printf("WARN ManagedDatabase watch disabled: both Kubernetes typed and dynamic clients are required") } diff --git a/components/control-plane/internal/gateway/namespace.go b/components/control-plane/internal/gateway/namespace.go index 52048210..f465de70 100644 --- a/components/control-plane/internal/gateway/namespace.go +++ b/components/control-plane/internal/gateway/namespace.go @@ -14,14 +14,20 @@ import ( "k8s.io/client-go/kubernetes" ) -// Labels createNamespace stamps on managed namespaces. Gateway namespace garbage -// collection sweeps managed namespaces whose names match the gateway prefix -// (openshell-) and excludes ManagedDatabase namespaces (openshell-db-). +// Labels EnsureManagedNamespace stamps on namespaces this control-plane instance +// owns. Periodic GC selects on the instance label so two HyperShells on one +// cluster never treat each other's gateway namespaces as orphans. const ( ManagedByLabel = "app.kubernetes.io/managed-by" ManagedByValue = "hypershell-control-plane" ManagedLabel = "hypershell.redhat.io/managed" ManagedLabelValue = "true" + // InstanceLabel identifies which control plane created the namespace. The + // value is unique to that controller: the namespace the controller pod + // runs in (HYPERSHELL_NAMESPACE, populated from the downward API). Periodic + // GC selects on this label so two HyperShells on one cluster never treat + // each other's gateways as orphans. + InstanceLabel = "hypershell.redhat.io/instance" // GatewayNamespacePrefix and DatabaseNamespacePrefix mirror the namespace names // the API server assigns in its BeforeCreate hooks (gatewayNamespacePrefix in @@ -45,36 +51,75 @@ const ( GCEligibleSinceAnnotation = "hypershell.redhat.io/gc-eligible-since" ) -// ManagedNamespaceSelector selects namespaces created by this control plane. -var ManagedNamespaceSelector = fmt.Sprintf("%s=%s,%s=%s", - ManagedLabel, ManagedLabelValue, ManagedByLabel, ManagedByValue) +// ManagedNamespaceSelector selects namespaces created by this control-plane +// instance. An empty instance matches nothing: listing by the generic management +// labels alone would treat every other HyperShell on the cluster as an orphan. +func ManagedNamespaceSelector(instance string) string { + if instance == "" { + return InstanceLabel + "=__no-such-instance__" + } + return fmt.Sprintf("%s=%s,%s=%s,%s=%s", + ManagedLabel, ManagedLabelValue, ManagedByLabel, ManagedByValue, InstanceLabel, instance) +} -// IsManagedNamespace reports whether a namespace was created and is managed by -// this control plane, based on the labels createNamespace stamps. -func IsManagedNamespace(ns *corev1.Namespace) bool { +// ManagedNamespaceLabels is the label set stamped on namespaces this instance +// creates and reconciles. +func ManagedNamespaceLabels(instance string) map[string]string { + return map[string]string{ + ManagedByLabel: ManagedByValue, + ManagedLabel: ManagedLabelValue, + InstanceLabel: instance, + } +} + +// hasManagementLabels reports whether ns carries the two generic HyperShell +// management labels, regardless of which instance created it. +func hasManagementLabels(ns *corev1.Namespace) bool { + if ns == nil { + return false + } return ns.Labels[ManagedLabel] == ManagedLabelValue && ns.Labels[ManagedByLabel] == ManagedByValue } +// IsManagedNamespace reports whether a namespace was created and is managed by +// this control-plane instance, based on the labels EnsureManagedNamespace stamps. +func IsManagedNamespace(ns *corev1.Namespace, instance string) bool { + if instance == "" || !hasManagementLabels(ns) { + return false + } + return ns.Labels[InstanceLabel] == instance +} + +// isGatewayWorkloadName reports whether name is a gateway workload namespace +// (openshell-) rather than a ManagedDatabase namespace (openshell-db-). +func isGatewayWorkloadName(name string) bool { + return strings.HasPrefix(name, GatewayNamespacePrefix) && + !strings.HasPrefix(name, DatabaseNamespacePrefix) +} + // IsGatewayNamespaceForGC reports whether ns is a gateway workload namespace -// subject to gateway namespace garbage collection. ManagedDatabase CNPG -// namespaces (openshell-db-*) carry the same management labels but are owned by -// the ManagedDatabase reconciler. Name-prefix matching keeps pre-existing orphaned -// gateway namespaces eligible for periodic GC without a label migration. -func IsGatewayNamespaceForGC(ns *corev1.Namespace) bool { - if !IsManagedNamespace(ns) { +// this control-plane instance owns and that periodic garbage collection may +// reap. ManagedDatabase CNPG namespaces (openshell-db-*) carry the same +// management labels but are owned by the ManagedDatabase reconciler. Namespaces +// owned by a different instance, or lacking this instance's identity label, are +// never eligible: another HyperShell's live gateways would otherwise look +// orphaned because they are absent from this instance's API server. +func IsGatewayNamespaceForGC(ns *corev1.Namespace, instance string) bool { + if !IsManagedNamespace(ns, instance) { return false } - return strings.HasPrefix(ns.Name, GatewayNamespacePrefix) && - !strings.HasPrefix(ns.Name, DatabaseNamespacePrefix) + return isGatewayWorkloadName(ns.Name) } // DeleteManagedNamespace deletes a gateway namespace, best-effort and -// idempotent. It only deletes namespaces subject to gateway namespace GC (see -// IsGatewayNamespaceForGC): an unmanaged, non-gateway, or already-absent -// namespace is treated as a no-op success. It returns deleted=true only when a -// delete call was issued. -func DeleteManagedNamespace(ctx context.Context, client kubernetes.Interface, namespace string) (bool, error) { +// idempotent. It only deletes namespaces this instance is allowed to remove: +// unmanaged, non-gateway, already-absent, or foreign-instance namespaces are +// treated as a no-op success. A legacy namespace that carries the two management +// labels but no instance label MAY be deleted, because this path is keyed to a +// Gateway from this instance's API server rather than a cluster-wide sweep. It +// returns deleted=true only when a delete call was issued. +func DeleteManagedNamespace(ctx context.Context, client kubernetes.Interface, namespace, instance string) (bool, error) { ns, err := client.CoreV1().Namespaces().Get(ctx, namespace, metav1.GetOptions{}) if err != nil { if k8serrors.IsNotFound(err) { @@ -83,10 +128,14 @@ func DeleteManagedNamespace(ctx context.Context, client kubernetes.Interface, na } return false, fmt.Errorf("get namespace %s: %w", namespace, err) } - if !IsGatewayNamespaceForGC(ns) { + if !isGatewayWorkloadName(ns.Name) || !hasManagementLabels(ns) { log.Printf("INFO namespace %s is not a gateway workload namespace, skipping deletion", namespace) return false, nil } + if labeled := ns.Labels[InstanceLabel]; labeled != "" && labeled != instance { + log.Printf("INFO namespace %s is managed by instance %q, not %q; skipping deletion", namespace, labeled, instance) + return false, nil + } if ns.DeletionTimestamp != nil { // Already terminating; nothing more to do. return false, nil @@ -101,6 +150,62 @@ func DeleteManagedNamespace(ctx context.Context, client kubernetes.Interface, na return true, nil } +// EnsureManagedNamespace creates namespace if it is absent, and otherwise +// reconciles the management and instance labels onto it. It refuses to adopt a +// namespace already labeled as a different control-plane instance. An empty +// instance is a configuration error: unlabeled namespaces are invisible to +// periodic GC and unsafe to claim on a shared cluster. +func EnsureManagedNamespace(ctx context.Context, client kubernetes.Interface, namespace, instance string) error { + if instance == "" { + return fmt.Errorf("refusing to manage namespace %s without a control-plane instance identity", namespace) + } + desired := ManagedNamespaceLabels(instance) + + ns, err := client.CoreV1().Namespaces().Get(ctx, namespace, metav1.GetOptions{}) + if k8serrors.IsNotFound(err) { + created := &corev1.Namespace{ + ObjectMeta: metav1.ObjectMeta{Name: namespace, Labels: desired}, + } + _, err = client.CoreV1().Namespaces().Create(ctx, created, metav1.CreateOptions{}) + if err == nil { + log.Printf("INFO created namespace %s (instance=%s)", namespace, instance) + return nil + } + if !k8serrors.IsAlreadyExists(err) { + return fmt.Errorf("create namespace %s: %w", namespace, err) + } + ns, err = client.CoreV1().Namespaces().Get(ctx, namespace, metav1.GetOptions{}) + if err != nil { + return fmt.Errorf("get namespace %s after create race: %w", namespace, err) + } + } else if err != nil { + return fmt.Errorf("get namespace %s: %w", namespace, err) + } + + if existing := ns.Labels[InstanceLabel]; existing != "" && existing != instance { + return fmt.Errorf("namespace %s is managed by instance %q, not %q", namespace, existing, instance) + } + + updated := ns.DeepCopy() + if updated.Labels == nil { + updated.Labels = map[string]string{} + } + changed := false + for k, v := range desired { + if updated.Labels[k] != v { + updated.Labels[k] = v + changed = true + } + } + if !changed { + return nil + } + if _, err := client.CoreV1().Namespaces().Update(ctx, updated, metav1.UpdateOptions{}); err != nil { + return fmt.Errorf("update namespace %s labels: %w", namespace, err) + } + return nil +} + // MarkGCEligible stamps the gc-eligible-since annotation with the given time if // it is not already present, returning the effective eligible-since time. The // timestamp is persisted on the namespace so the grace period survives diff --git a/components/control-plane/internal/gateway/namespace_test.go b/components/control-plane/internal/gateway/namespace_test.go index 6b61605c..7cbb0d24 100644 --- a/components/control-plane/internal/gateway/namespace_test.go +++ b/components/control-plane/internal/gateway/namespace_test.go @@ -12,34 +12,57 @@ import ( ) func managedNamespace(name string, annotations map[string]string) *corev1.Namespace { + return managedNamespaceForInstance(name, "hypershell", annotations) +} + +func managedNamespaceForInstance(name, instance string, annotations map[string]string) *corev1.Namespace { + labels := map[string]string{ + ManagedByLabel: ManagedByValue, + ManagedLabel: ManagedLabelValue, + } + if instance != "" { + labels[InstanceLabel] = instance + } return &corev1.Namespace{ ObjectMeta: metav1.ObjectMeta{ - Name: name, - Labels: map[string]string{ - ManagedByLabel: ManagedByValue, - ManagedLabel: ManagedLabelValue, - }, + Name: name, + Labels: labels, Annotations: annotations, }, } } +func TestManagedNamespaceSelector(t *testing.T) { + got := ManagedNamespaceSelector("alice") + want := "hypershell.redhat.io/managed=true,app.kubernetes.io/managed-by=hypershell-control-plane,hypershell.redhat.io/instance=alice" + if got != want { + t.Errorf("ManagedNamespaceSelector() = %q, want %q", got, want) + } + if ManagedNamespaceSelector("") == ManagedNamespaceSelector("hypershell") { + t.Errorf("empty instance selector must not match a real instance") + } +} + func TestIsManagedNamespace(t *testing.T) { tests := []struct { - name string - labels map[string]string - want bool + name string + labels map[string]string + instance string + want bool }{ - {"both labels", map[string]string{ManagedByLabel: ManagedByValue, ManagedLabel: ManagedLabelValue}, true}, - {"missing managed-by", map[string]string{ManagedLabel: ManagedLabelValue}, false}, - {"missing managed", map[string]string{ManagedByLabel: ManagedByValue}, false}, - {"wrong managed-by value", map[string]string{ManagedByLabel: "someone-else", ManagedLabel: ManagedLabelValue}, false}, - {"no labels", nil, false}, + {"this instance", map[string]string{ManagedByLabel: ManagedByValue, ManagedLabel: ManagedLabelValue, InstanceLabel: "hypershell"}, "hypershell", true}, + {"foreign instance", map[string]string{ManagedByLabel: ManagedByValue, ManagedLabel: ManagedLabelValue, InstanceLabel: "stage"}, "hypershell", false}, + {"missing instance label", map[string]string{ManagedByLabel: ManagedByValue, ManagedLabel: ManagedLabelValue}, "hypershell", false}, + {"empty instance identity", map[string]string{ManagedByLabel: ManagedByValue, ManagedLabel: ManagedLabelValue, InstanceLabel: "hypershell"}, "", false}, + {"missing managed-by", map[string]string{ManagedLabel: ManagedLabelValue, InstanceLabel: "hypershell"}, "hypershell", false}, + {"missing managed", map[string]string{ManagedByLabel: ManagedByValue, InstanceLabel: "hypershell"}, "hypershell", false}, + {"wrong managed-by value", map[string]string{ManagedByLabel: "someone-else", ManagedLabel: ManagedLabelValue, InstanceLabel: "hypershell"}, "hypershell", false}, + {"no labels", nil, "hypershell", false}, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { ns := &corev1.Namespace{ObjectMeta: metav1.ObjectMeta{Name: "x", Labels: tt.labels}} - if got := IsManagedNamespace(ns); got != tt.want { + if got := IsManagedNamespace(ns, tt.instance); got != tt.want { t.Errorf("IsManagedNamespace() = %v, want %v", got, tt.want) } }) @@ -48,25 +71,31 @@ func TestIsManagedNamespace(t *testing.T) { func TestIsGatewayNamespaceForGC(t *testing.T) { tests := []struct { - name string - ns string - want bool + name string + ns string + instance string + labels map[string]string + want bool }{ - {"gateway namespace", "openshell-a14873d1631f1b74", true}, - {"e2e orphan", "openshell-e2e-orphan-123", true}, - // A gateway hash may begin with the hex letters "db"; the trailing dash in - // the database prefix keeps it classified as a gateway namespace. - {"gateway hash starting with db", "openshell-db1a2b3c4d5e6f70", true}, - {"managed database namespace", "openshell-db-a1b2c3d4e5f67890", false}, - {"unmanaged", "openshell-gw", false}, + {"this instance gateway", "openshell-a14873d1631f1b74", "hypershell", nil, true}, + {"e2e orphan", "openshell-e2e-orphan-123", "hypershell", nil, true}, + {"gateway hash starting with db", "openshell-db1a2b3c4d5e6f70", "hypershell", nil, true}, + {"managed database namespace", "openshell-db-a1b2c3d4e5f67890", "hypershell", nil, false}, + {"foreign instance", "openshell-a14873d1631f1b74", "hypershell", map[string]string{ + ManagedByLabel: ManagedByValue, ManagedLabel: ManagedLabelValue, InstanceLabel: "stage", + }, false}, + {"unlabeled legacy", "openshell-a14873d1631f1b74", "hypershell", map[string]string{ + ManagedByLabel: ManagedByValue, ManagedLabel: ManagedLabelValue, + }, false}, + {"unmanaged", "openshell-gw", "hypershell", map[string]string{}, false}, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { ns := managedNamespace(tt.ns, nil) - if tt.name == "unmanaged" { - ns.Labels = nil + if tt.labels != nil { + ns.Labels = tt.labels } - if got := IsGatewayNamespaceForGC(ns); got != tt.want { + if got := IsGatewayNamespaceForGC(ns, tt.instance); got != tt.want { t.Errorf("IsGatewayNamespaceForGC() = %v, want %v", got, tt.want) } }) @@ -78,7 +107,7 @@ func TestDeleteManagedNamespace(t *testing.T) { t.Run("deletes a managed namespace", func(t *testing.T) { client := fake.NewSimpleClientset(managedNamespace("openshell-gw", nil)) - deleted, err := DeleteManagedNamespace(ctx, client, "openshell-gw") + deleted, err := DeleteManagedNamespace(ctx, client, "openshell-gw", "hypershell") if err != nil { t.Fatalf("DeleteManagedNamespace() error = %v", err) } @@ -93,7 +122,7 @@ func TestDeleteManagedNamespace(t *testing.T) { t.Run("skips an unmanaged namespace", func(t *testing.T) { unmanaged := &corev1.Namespace{ObjectMeta: metav1.ObjectMeta{Name: "shared"}} client := fake.NewSimpleClientset(unmanaged) - deleted, err := DeleteManagedNamespace(ctx, client, "shared") + deleted, err := DeleteManagedNamespace(ctx, client, "shared", "hypershell") if err != nil { t.Fatalf("DeleteManagedNamespace() error = %v", err) } @@ -107,7 +136,7 @@ func TestDeleteManagedNamespace(t *testing.T) { t.Run("skips a managed database namespace", func(t *testing.T) { client := fake.NewSimpleClientset(managedNamespace("openshell-db-a1b2c3d4e5f67890", nil)) - deleted, err := DeleteManagedNamespace(ctx, client, "openshell-db-a1b2c3d4e5f67890") + deleted, err := DeleteManagedNamespace(ctx, client, "openshell-db-a1b2c3d4e5f67890", "hypershell") if err != nil { t.Fatalf("DeleteManagedNamespace() error = %v", err) } @@ -121,7 +150,7 @@ func TestDeleteManagedNamespace(t *testing.T) { t.Run("absent namespace is a no-op success", func(t *testing.T) { client := fake.NewSimpleClientset() - deleted, err := DeleteManagedNamespace(ctx, client, "gone") + deleted, err := DeleteManagedNamespace(ctx, client, "gone", "hypershell") if err != nil { t.Fatalf("DeleteManagedNamespace() error = %v", err) } @@ -129,6 +158,87 @@ func TestDeleteManagedNamespace(t *testing.T) { t.Errorf("deleted = true, want false for absent namespace") } }) + + t.Run("skips a foreign instance namespace", func(t *testing.T) { + client := fake.NewSimpleClientset(managedNamespaceForInstance("openshell-gw", "stage", nil)) + deleted, err := DeleteManagedNamespace(ctx, client, "openshell-gw", "hypershell") + if err != nil { + t.Fatalf("DeleteManagedNamespace() error = %v", err) + } + if deleted { + t.Errorf("deleted = true, want false for foreign instance") + } + if _, err := client.CoreV1().Namespaces().Get(ctx, "openshell-gw", metav1.GetOptions{}); err != nil { + t.Errorf("foreign instance namespace should be preserved, err = %v", err) + } + }) + + t.Run("deletes a legacy unlabeled managed gateway namespace", func(t *testing.T) { + client := fake.NewSimpleClientset(managedNamespaceForInstance("openshell-gw", "", nil)) + deleted, err := DeleteManagedNamespace(ctx, client, "openshell-gw", "hypershell") + if err != nil { + t.Fatalf("DeleteManagedNamespace() error = %v", err) + } + if !deleted { + t.Errorf("deleted = false, want true for unlabeled legacy namespace on delete-driven path") + } + }) +} + +func TestEnsureManagedNamespace(t *testing.T) { + ctx := context.Background() + + t.Run("creates with instance label", func(t *testing.T) { + client := fake.NewSimpleClientset() + if err := EnsureManagedNamespace(ctx, client, "openshell-gw", "hypershell"); err != nil { + t.Fatalf("EnsureManagedNamespace() error = %v", err) + } + got, err := client.CoreV1().Namespaces().Get(ctx, "openshell-gw", metav1.GetOptions{}) + if err != nil { + t.Fatalf("get namespace: %v", err) + } + if got.Labels[InstanceLabel] != "hypershell" { + t.Errorf("instance label = %q, want hypershell", got.Labels[InstanceLabel]) + } + if got.Labels[ManagedLabel] != ManagedLabelValue || got.Labels[ManagedByLabel] != ManagedByValue { + t.Errorf("management labels missing: %v", got.Labels) + } + }) + + t.Run("stamps instance on unlabeled legacy namespace", func(t *testing.T) { + client := fake.NewSimpleClientset(managedNamespaceForInstance("openshell-gw", "", nil)) + if err := EnsureManagedNamespace(ctx, client, "openshell-gw", "hypershell"); err != nil { + t.Fatalf("EnsureManagedNamespace() error = %v", err) + } + got, err := client.CoreV1().Namespaces().Get(ctx, "openshell-gw", metav1.GetOptions{}) + if err != nil { + t.Fatalf("get namespace: %v", err) + } + if got.Labels[InstanceLabel] != "hypershell" { + t.Errorf("instance label = %q, want hypershell after reconcile", got.Labels[InstanceLabel]) + } + }) + + t.Run("refuses a foreign instance namespace", func(t *testing.T) { + client := fake.NewSimpleClientset(managedNamespaceForInstance("openshell-gw", "stage", nil)) + if err := EnsureManagedNamespace(ctx, client, "openshell-gw", "hypershell"); err == nil { + t.Fatalf("EnsureManagedNamespace() error = nil, want foreign instance error") + } + got, err := client.CoreV1().Namespaces().Get(ctx, "openshell-gw", metav1.GetOptions{}) + if err != nil { + t.Fatalf("get namespace: %v", err) + } + if got.Labels[InstanceLabel] != "stage" { + t.Errorf("instance label overwritten to %q, want stage", got.Labels[InstanceLabel]) + } + }) + + t.Run("refuses an empty instance identity", func(t *testing.T) { + client := fake.NewSimpleClientset() + if err := EnsureManagedNamespace(ctx, client, "openshell-gw", ""); err == nil { + t.Fatalf("EnsureManagedNamespace() error = nil, want empty instance error") + } + }) } func TestMarkGCEligible(t *testing.T) { diff --git a/components/control-plane/internal/gateway/reconciler.go b/components/control-plane/internal/gateway/reconciler.go index 353f4e30..8e60dcfd 100644 --- a/components/control-plane/internal/gateway/reconciler.go +++ b/components/control-plane/internal/gateway/reconciler.go @@ -59,10 +59,8 @@ func ReconcileGateway( } ingressMode := gatewayIngressMode(opts) - if !namespaceExists(ctx, clientset, nsConfig.Name) { - if err := createNamespace(ctx, clientset, nsConfig.Name); err != nil { - return fmt.Errorf("create namespace %s: %w", nsConfig.Name, err) - } + if err := EnsureManagedNamespace(ctx, clientset, nsConfig.Name, opts.ControlPlaneNamespace); err != nil { + return fmt.Errorf("ensure namespace %s: %w", nsConfig.Name, err) } if err := ValidateGatewayConfig(nsConfig.Gateway); err != nil { @@ -766,38 +764,6 @@ func DeleteRouteResources(ctx context.Context, dynamicClient dynamic.Interface, return errors.Join(errs...) } -func NamespaceExists(ctx context.Context, clientset kubernetes.Interface, namespace string) bool { - _, err := clientset.CoreV1().Namespaces().Get(ctx, namespace, metav1.GetOptions{}) - return err == nil -} - -func namespaceExists(ctx context.Context, clientset *kubernetes.Clientset, namespace string) bool { - return NamespaceExists(ctx, clientset, namespace) -} - -func CreateManagedNamespace(ctx context.Context, clientset kubernetes.Interface, namespace string) error { - return createNamespace(ctx, clientset, namespace) -} - -func createNamespace(ctx context.Context, clientset kubernetes.Interface, namespace string) error { - ns := &corev1.Namespace{ - ObjectMeta: metav1.ObjectMeta{ - Name: namespace, - Labels: map[string]string{ - ManagedByLabel: ManagedByValue, - ManagedLabel: ManagedLabelValue, - }, - }, - } - - _, err := clientset.CoreV1().Namespaces().Create(ctx, ns, metav1.CreateOptions{}) - if err != nil && !k8serrors.IsAlreadyExists(err) { - return fmt.Errorf("create namespace: %w", err) - } - log.Printf("INFO created namespace %s", namespace) - return nil -} - func deployGateway( ctx context.Context, dynamicClient dynamic.Interface, diff --git a/components/control-plane/internal/reconciler/managed_database_lifecycle_test.go b/components/control-plane/internal/reconciler/managed_database_lifecycle_test.go index 1c6bbfd0..fce72e3b 100644 --- a/components/control-plane/internal/reconciler/managed_database_lifecycle_test.go +++ b/components/control-plane/internal/reconciler/managed_database_lifecycle_test.go @@ -29,7 +29,7 @@ func TestManagedDatabaseDeleteUsesTombstoneAndIsIdempotent(t *testing.T) { const namespace = "database-ns" dynamic := dynamicfake.NewSimpleDynamicClient(runtime.NewScheme(), deploymentCleanupObjects(namespace)...) typed := kubernetesfake.NewSimpleClientset(&corev1.Namespace{ObjectMeta: metav1.ObjectMeta{Name: namespace}}) - r := NewManagedDatabaseReconciler(dynamic, typed, nil) + r := NewManagedDatabaseReconciler(dynamic, typed, nil, "hypershell") event := watcher.Event[*pb.ManagedDatabase]{Type: watcher.EventDeleted, ResourceID: "db-1", Resource: &pb.ManagedDatabase{Name: "database", Namespace: namespace, Provider: "deployment"}} if err := r.Handle(context.Background(), event); err != nil { t.Fatalf("delete: %v", err) @@ -50,7 +50,7 @@ func TestManagedDatabaseDeleteNilTombstoneUsesLastSeenAndRetainsOnFailure(t *tes const namespace = "database-ns" dynamic := dynamicfake.NewSimpleDynamicClient(runtime.NewScheme(), deploymentCleanupObjects(namespace)...) typed := kubernetesfake.NewSimpleClientset(&corev1.Namespace{ObjectMeta: metav1.ObjectMeta{Name: namespace}}) - r := NewManagedDatabaseReconciler(dynamic, typed, nil) + r := NewManagedDatabaseReconciler(dynamic, typed, nil, "hypershell") fail := true dynamic.PrependReactor("delete", "deployments", func(k8stesting.Action) (bool, runtime.Object, error) { if fail { @@ -76,7 +76,7 @@ func TestManagedDatabaseDeleteNilTombstoneUsesLastSeenAndRetainsOnFailure(t *tes } func TestManagedDatabaseReconcilerNilClientsReturnsError(t *testing.T) { - r := NewManagedDatabaseReconciler(nil, nil, nil) + r := NewManagedDatabaseReconciler(nil, nil, nil, "") err := r.Handle(context.Background(), watcher.Event[*pb.ManagedDatabase]{Type: watcher.EventDeleted, ResourceID: "db-1", Resource: &pb.ManagedDatabase{Namespace: "must-not-guess", Provider: "deployment"}}) if err == nil { t.Fatal("want nil client error") @@ -86,7 +86,7 @@ func TestManagedDatabaseReconcilerNilClientsReturnsError(t *testing.T) { func TestDeleteCNPGClusterPropagatesErrorsAndNotFoundIsIdempotent(t *testing.T) { dynamic := dynamicfake.NewSimpleDynamicClient(runtime.NewScheme()) typed := kubernetesfake.NewSimpleClientset() - r := NewManagedDatabaseReconciler(dynamic, typed, nil) + r := NewManagedDatabaseReconciler(dynamic, typed, nil, "hypershell") dynamic.PrependReactor("delete", "clusters", func(k8stesting.Action) (bool, runtime.Object, error) { return true, nil, errors.New("CNPG unavailable") }) diff --git a/components/control-plane/internal/reconciler/managed_database_test.go b/components/control-plane/internal/reconciler/managed_database_test.go index 539769dd..b96eb5ef 100644 --- a/components/control-plane/internal/reconciler/managed_database_test.go +++ b/components/control-plane/internal/reconciler/managed_database_test.go @@ -21,7 +21,7 @@ import ( ) func TestNewManagedDatabaseReconcilerWithoutKubernetesClient(t *testing.T) { - r := NewManagedDatabaseReconciler(nil, nil, nil) + r := NewManagedDatabaseReconciler(nil, nil, nil, "") if r.hasCNPG { t.Fatal("hasCNPG = true without a Kubernetes client, want false") } @@ -223,7 +223,7 @@ func TestReconcileDeploymentDatabaseUsesOpenShellImageAndPostgresSecurityContext clientset := kubernetesfake.NewSimpleClientset() dynamicClient := dynamicfake.NewSimpleDynamicClient(runtime.NewScheme()) - r := &ManagedDatabaseReconciler{clientset: clientset, dynamicClient: dynamicClient} + r := &ManagedDatabaseReconciler{clientset: clientset, dynamicClient: dynamicClient, controlPlaneNamespace: "hypershell"} ctx, cancel := context.WithCancel(context.Background()) cancel() // Avoid waiting for fake Deployment readiness after resources are applied. if err := r.reconcileDeploymentDatabase(ctx, &pb.ManagedDatabase{Namespace: namespace}); !errors.Is(err, context.Canceled) { diff --git a/components/control-plane/internal/reconciler/namespace.go b/components/control-plane/internal/reconciler/namespace.go index 6473ac1e..5b3ec843 100644 --- a/components/control-plane/internal/reconciler/namespace.go +++ b/components/control-plane/internal/reconciler/namespace.go @@ -42,7 +42,9 @@ const ( ) // NamespaceGCReconciler periodically garbage-collects gateway namespaces that -// the control plane created but that no longer have a live Gateway backing them. +// this control-plane instance created but that no longer have a live Gateway in +// this instance's API server. Other HyperShell instances on the same cluster are +// ignored: the sweep selects on hypershell.redhat.io/instance=. // This reaps namespaces orphaned by a delete event missed while the control // plane was down, and namespaces whose gateway failed to bootstrap and was then // deleted. Reaping is best-effort and idempotent, and is delayed by a grace @@ -91,7 +93,7 @@ func NewNamespaceGCReconciler(client kubernetes.Interface, grpcConn *grpc.Client // Run drives the garbage-collection loop until the context is cancelled. func (r *NamespaceGCReconciler) Run(ctx context.Context) error { - log.Printf("INFO namespace GC reconciler started (interval=%s grace=%s)", r.interval, r.gracePeriod) + log.Printf("INFO namespace GC reconciler started (interval=%s grace=%s instance=%s)", r.interval, r.gracePeriod, r.cpNamespace) r.reconcileOnce(ctx) ticker := time.NewTicker(r.interval) defer ticker.Stop() @@ -111,6 +113,15 @@ func (r *NamespaceGCReconciler) reconcileOnce(ctx context.Context) { var tickErr error defer func() { endSpan(tickErr) }() + // An empty instance identity would list every HyperShell-managed namespace + // on the cluster. Abort rather than treat another instance's gateways as + // orphans of this one. + if r.cpNamespace == "" { + tickErr = fmt.Errorf("no control-plane namespace configured; refusing to sweep") + log.Printf("WARN namespace gc: %v", tickErr) + return + } + // Build the set of namespaces backed by a live Gateway. If we cannot list // gateways we must abort the whole sweep: an empty or failed list would make // every managed namespace look orphaned and risk reaping live ones. @@ -123,7 +134,7 @@ func (r *NamespaceGCReconciler) reconcileOnce(ctx context.Context) { listCtx, cancel := context.WithTimeout(ctx, namespaceListTimeout) namespaces, err := r.client.CoreV1().Namespaces().List(listCtx, metav1.ListOptions{ - LabelSelector: gateway.ManagedNamespaceSelector, + LabelSelector: gateway.ManagedNamespaceSelector(r.cpNamespace), }) cancel() if err != nil { @@ -179,7 +190,7 @@ func (r *NamespaceGCReconciler) grpcLiveNamespaces(ctx context.Context) (map[str func (r *NamespaceGCReconciler) reconcileNamespace(ctx context.Context, ns *corev1.Namespace, live map[string]struct{}) error { // Defense in depth: only gateway workload namespaces are subject to this // reconciler, even if the server-side label selector over-returns. - if !gateway.IsGatewayNamespaceForGC(ns) { + if !gateway.IsGatewayNamespaceForGC(ns, r.cpNamespace) { return nil } // A namespace already terminating needs no further action. @@ -255,7 +266,7 @@ func (r *NamespaceGCReconciler) reconcileNamespace(ctx context.Context, ns *core deleteCtx, cancel := context.WithTimeout(ctx, namespaceOperationTimeout) defer cancel() - if _, err := gateway.DeleteManagedNamespace(deleteCtx, r.client, ns.Name); err != nil { + if _, err := gateway.DeleteManagedNamespace(deleteCtx, r.client, ns.Name, r.cpNamespace); err != nil { return err } return nil diff --git a/components/control-plane/internal/reconciler/namespace_test.go b/components/control-plane/internal/reconciler/namespace_test.go index 5b1e4394..df3fa386 100644 --- a/components/control-plane/internal/reconciler/namespace_test.go +++ b/components/control-plane/internal/reconciler/namespace_test.go @@ -21,12 +21,23 @@ func managedNS(name string, annotations map[string]string) *corev1.Namespace { Labels: map[string]string{ gateway.ManagedByLabel: gateway.ManagedByValue, gateway.ManagedLabel: gateway.ManagedLabelValue, + gateway.InstanceLabel: "hypershell", }, Annotations: annotations, }, } } +func nsWithLabels(name string, labels, annotations map[string]string) *corev1.Namespace { + return &corev1.Namespace{ + ObjectMeta: metav1.ObjectMeta{ + Name: name, + Labels: labels, + Annotations: annotations, + }, + } +} + // newTestGC builds a NamespaceGCReconciler with a fixed clock and no gRPC // connection. reconcileNamespace re-confirms liveness through the liveNamespaces // seam immediately before a delete, so the seam is defaulted here to "nothing is @@ -287,3 +298,103 @@ func TestReconcileNamespace_UnmanagedIsUntouched(t *testing.T) { t.Errorf("unmanaged namespace was annotated, want untouched") } } + +func TestReconcileOnce_OnlySweepsThisInstance(t *testing.T) { + ctx := context.Background() + now := time.Date(2026, 8, 17, 12, 0, 0, 0, time.UTC) + own := managedNS("openshell-own", nil) + foreign := nsWithLabels("openshell-stage", map[string]string{ + gateway.ManagedByLabel: gateway.ManagedByValue, + gateway.ManagedLabel: gateway.ManagedLabelValue, + gateway.InstanceLabel: "hypershell-stage", + }, nil) + unlabeled := nsWithLabels("openshell-legacy", map[string]string{ + gateway.ManagedByLabel: gateway.ManagedByValue, + gateway.ManagedLabel: gateway.ManagedLabelValue, + }, nil) + client := fake.NewSimpleClientset(own, foreign, unlabeled) + r := newTestGC(client, now) + + r.reconcileOnce(ctx) + + ownUpdated, err := client.CoreV1().Namespaces().Get(ctx, "openshell-own", metav1.GetOptions{}) + if err != nil { + t.Fatalf("get own namespace: %v", err) + } + if ownUpdated.Annotations[gateway.GCEligibleSinceAnnotation] != now.Format(time.RFC3339) { + t.Errorf("this instance's orphan was not stamped gc-eligible-since") + } + + for _, name := range []string{"openshell-stage", "openshell-legacy"} { + got, err := client.CoreV1().Namespaces().Get(ctx, name, metav1.GetOptions{}) + if err != nil { + t.Fatalf("get %s: %v", name, err) + } + if _, ok := got.Annotations[gateway.GCEligibleSinceAnnotation]; ok { + t.Errorf("%s was treated as an orphan of this instance, want ignored", name) + } + } +} + +func TestReconcileOnce_DoesNotReapForeignInstancePastGrace(t *testing.T) { + ctx := context.Background() + now := time.Date(2026, 8, 17, 12, 0, 0, 0, time.UTC) + foreign := nsWithLabels("openshell-stage", map[string]string{ + gateway.ManagedByLabel: gateway.ManagedByValue, + gateway.ManagedLabel: gateway.ManagedLabelValue, + gateway.InstanceLabel: "hypershell-stage", + }, map[string]string{ + gateway.GCEligibleSinceAnnotation: now.Add(-20 * time.Minute).Format(time.RFC3339), + }) + client := fake.NewSimpleClientset(foreign) + r := newTestGC(client, now) + + r.reconcileOnce(ctx) + + if !nsExists(t, client, "openshell-stage") { + t.Fatalf("foreign instance namespace reaped, want retained") + } +} + +func TestReconcileNamespace_SkipsForeignInstance(t *testing.T) { + ctx := context.Background() + now := time.Date(2026, 8, 17, 12, 0, 0, 0, time.UTC) + foreign := nsWithLabels("openshell-stage", map[string]string{ + gateway.ManagedByLabel: gateway.ManagedByValue, + gateway.ManagedLabel: gateway.ManagedLabelValue, + gateway.InstanceLabel: "hypershell-stage", + }, nil) + client := fake.NewSimpleClientset(foreign) + r := newTestGC(client, now) + + if err := r.reconcileNamespace(ctx, foreign, map[string]struct{}{}); err != nil { + t.Fatalf("reconcileNamespace() error = %v", err) + } + + updated, err := client.CoreV1().Namespaces().Get(ctx, "openshell-stage", metav1.GetOptions{}) + if err != nil { + t.Fatalf("get namespace: %v", err) + } + if _, ok := updated.Annotations[gateway.GCEligibleSinceAnnotation]; ok { + t.Errorf("foreign instance namespace was annotated, want untouched") + } +} + +func TestReconcileOnce_EmptyInstanceAbortsSweep(t *testing.T) { + ctx := context.Background() + now := time.Date(2026, 8, 17, 12, 0, 0, 0, time.UTC) + ns := managedNS("openshell-gw", nil) + client := fake.NewSimpleClientset(ns) + r := newTestGC(client, now) + r.cpNamespace = "" + + r.reconcileOnce(ctx) + + updated, err := client.CoreV1().Namespaces().Get(ctx, "openshell-gw", metav1.GetOptions{}) + if err != nil { + t.Fatalf("get namespace: %v", err) + } + if _, ok := updated.Annotations[gateway.GCEligibleSinceAnnotation]; ok { + t.Errorf("sweep with empty instance annotated a namespace, want aborted") + } +} diff --git a/components/control-plane/internal/reconciler/reconciler.go b/components/control-plane/internal/reconciler/reconciler.go index 89678ee0..20aec087 100644 --- a/components/control-plane/internal/reconciler/reconciler.go +++ b/components/control-plane/internal/reconciler/reconciler.go @@ -71,21 +71,23 @@ func (r *ManagedClusterReconciler) Handle(ctx context.Context, event watcher.Eve } type ManagedDatabaseReconciler struct { - mu sync.Mutex - active map[string]struct{} - pending map[string]watcher.Event[*pb.ManagedDatabase] - dynamicClient dynamic.Interface - clientset kubernetes.Interface - grpcConn *grpc.ClientConn - hasCNPG bool - isOpenShift bool - lastSeen map[string]*pb.ManagedDatabase + mu sync.Mutex + active map[string]struct{} + pending map[string]watcher.Event[*pb.ManagedDatabase] + dynamicClient dynamic.Interface + clientset kubernetes.Interface + grpcConn *grpc.ClientConn + hasCNPG bool + isOpenShift bool + controlPlaneNamespace string + lastSeen map[string]*pb.ManagedDatabase } func NewManagedDatabaseReconciler( dynamicClient dynamic.Interface, clientset kubernetes.Interface, grpcConn *grpc.ClientConn, + controlPlaneNamespace string, ) *ManagedDatabaseReconciler { hasCNPG := false isOpenShift := false @@ -98,14 +100,15 @@ func NewManagedDatabaseReconciler( } } return &ManagedDatabaseReconciler{ - active: make(map[string]struct{}), - pending: make(map[string]watcher.Event[*pb.ManagedDatabase]), - lastSeen: make(map[string]*pb.ManagedDatabase), - dynamicClient: dynamicClient, - clientset: clientset, - grpcConn: grpcConn, - hasCNPG: hasCNPG, - isOpenShift: isOpenShift, + active: make(map[string]struct{}), + pending: make(map[string]watcher.Event[*pb.ManagedDatabase]), + lastSeen: make(map[string]*pb.ManagedDatabase), + dynamicClient: dynamicClient, + clientset: clientset, + grpcConn: grpcConn, + hasCNPG: hasCNPG, + isOpenShift: isOpenShift, + controlPlaneNamespace: controlPlaneNamespace, } } @@ -304,10 +307,8 @@ func (r *ManagedDatabaseReconciler) handleDeploymentDatabase(ctx context.Context func (r *ManagedDatabaseReconciler) reconcileCNPGCluster(ctx context.Context, db *pb.ManagedDatabase) error { namespace := db.Namespace - if !gateway.NamespaceExists(ctx, r.clientset, namespace) { - if err := gateway.CreateManagedNamespace(ctx, r.clientset, namespace); err != nil { - return fmt.Errorf("create namespace %s: %w", namespace, err) - } + if err := gateway.EnsureManagedNamespace(ctx, r.clientset, namespace, r.controlPlaneNamespace); err != nil { + return fmt.Errorf("ensure namespace %s: %w", namespace, err) } clusterName := managedDatabaseCNPGClusterName() @@ -468,31 +469,7 @@ func (r *ManagedDatabaseReconciler) deleteCNPGCluster(ctx context.Context, names return errors.Join(errs...) } func (r *ManagedDatabaseReconciler) reconcileDeploymentDatabaseNamespace(ctx context.Context, namespace string) error { - namespaces := r.clientset.CoreV1().Namespaces() - existing, err := namespaces.Get(ctx, namespace, metav1.GetOptions{}) - if k8serrors.IsNotFound(err) { - if err := gateway.CreateManagedNamespace(ctx, r.clientset, namespace); err != nil { - return fmt.Errorf("create namespace %s: %w", namespace, err) - } - return nil - } - if err != nil { - return fmt.Errorf("get namespace %s: %w", namespace, err) - } - - updated := existing.DeepCopy() - if updated.Labels == nil { - updated.Labels = map[string]string{} - } - updated.Labels[gateway.ManagedByLabel] = gateway.ManagedByValue - updated.Labels[gateway.ManagedLabel] = gateway.ManagedLabelValue - if reflect.DeepEqual(existing.Labels, updated.Labels) { - return nil - } - if _, err := namespaces.Update(ctx, updated, metav1.UpdateOptions{}); err != nil { - return fmt.Errorf("update namespace %s labels: %w", namespace, err) - } - return nil + return gateway.EnsureManagedNamespace(ctx, r.clientset, namespace, r.controlPlaneNamespace) } func (r *ManagedDatabaseReconciler) reconcileDeploymentDatabaseCredentials(ctx context.Context, namespace, name string) error { @@ -1347,7 +1324,7 @@ func (r *GatewayReconciler) Handle(ctx context.Context, event watcher.Event[*pb. // Namespace deletion and database deletion are independent cleanup // operations. Attempt both and aggregate failures so one partial failure // cannot silently leak the other resource. - deleted, err := gateway.DeleteManagedNamespace(ctx, r.clientset, namespace) + deleted, err := gateway.DeleteManagedNamespace(ctx, r.clientset, namespace, r.controlPlaneNamespace) if err != nil { deleteErrs = append(deleteErrs, fmt.Errorf("delete gateway namespace %s: %w", namespace, err)) } else if !deleted { diff --git a/deploy/base/controller.yaml b/deploy/base/controller.yaml index efea84f0..acfa419d 100644 --- a/deploy/base/controller.yaml +++ b/deploy/base/controller.yaml @@ -38,7 +38,9 @@ spec: - name: HYPERSHELL_API_SERVER_URL value: "http://hypershell-api-server:8000" - name: HYPERSHELL_NAMESPACE - value: "hypershell-system" + valueFrom: + fieldRef: + fieldPath: metadata.namespace - name: HYPERSHELL_LOG_LEVEL value: "info" # Plaintext in-cluster gRPC provisioner endpoint. A NetworkPolicy diff --git a/scripts/cluster/drivers/kind.sh b/scripts/cluster/drivers/kind.sh index 6e60ff8a..c6648ac8 100755 --- a/scripts/cluster/drivers/kind.sh +++ b/scripts/cluster/drivers/kind.sh @@ -22,6 +22,10 @@ cluster_status() { exec "${KIND_SCRIPTS}/status.sh" } +cluster_seed() { + exec "${KIND_SCRIPTS}/seed.sh" +} + component_swap() { exec "${KIND_SCRIPTS}/swap-component.sh" up "$1" } diff --git a/scripts/cluster/drivers/openshift.sh b/scripts/cluster/drivers/openshift.sh index 742b3397..a50cc677 100755 --- a/scripts/cluster/drivers/openshift.sh +++ b/scripts/cluster/drivers/openshift.sh @@ -786,6 +786,10 @@ seed_via_api() { local i if ! command -v curl >/dev/null 2>&1; then warn "curl is required on this machine to seed the API; skip automatic seeding" + if seed_strict; then + error "Platform seeding failed and SEED_STRICT=true - failing" + return 1 + fi return 0 fi info "Obtaining API token from Keycloak Route..." @@ -801,6 +805,10 @@ seed_via_api() { done if [[ -z "${token}" ]]; then warn "Could not obtain API token; skip automatic seeding" + if seed_strict; then + error "Platform seeding failed and SEED_STRICT=true - failing" + return 1 + fi return 0 fi success "API token obtained" @@ -937,6 +945,15 @@ seed_via_api() { if [[ -n "${seed_failed}" ]]; then warn "Automatic seeding incomplete - create resources manually after the API server is ready" + warn "Hint: If your local branch has schema / contract changes (e.g. dropped 'fleet_id'), the baseline API server image in the registry might reject your seed request." + warn " To resolve this, swap in your working-tree API server build first, then run 'make openshift-seed':" + warn " 1. SKIP_SEED=true make openshift-up" + warn " 2. make openshift-api-server-up" + warn " 3. make openshift-seed" + if seed_strict; then + error "Platform seeding failed and SEED_STRICT=true - failing" + return 1 + fi fi } @@ -978,11 +995,27 @@ cluster_up() { configure_oidc_from_routes wait_for_deployments add_keycloak_redirect_uri || true - seed_via_api + if skip_seed; then + info "SKIP_SEED=true - skipping platform seeding" + else + seed_via_api + fi echo "" print_banner } +cluster_seed() { + header "Seeding HyperShell on OpenShift" + echo "" + require_openshift_cluster + resolve_openshift_namespace + OPENSHIFT_ENTRY_PROJECT="$(current_project)" + trap 'if [[ -n "${OPENSHIFT_ENTRY_PROJECT:-}" ]]; then oc_cli project "${OPENSHIFT_ENTRY_PROJECT}" >/dev/null 2>&1 || true; fi' EXIT + validate_namespace_group + configure_oidc_from_routes + seed_via_api +} + verify_owned_namespace() { local ns="$1" if ! namespace_exists "${ns}"; then diff --git a/scripts/cluster/lib.sh b/scripts/cluster/lib.sh index 58668fc1..0605e6f4 100755 --- a/scripts/cluster/lib.sh +++ b/scripts/cluster/lib.sh @@ -36,7 +36,7 @@ REPO_ROOT="$(cd "${CLUSTER_SCRIPT_DIR}/../.." && pwd)" : "${web_console_local:=localhost/hypershell-web-console:dev}" : "${build_version:=$(git -C "${REPO_ROOT}" rev-parse --short HEAD 2>/dev/null || echo unknown)}" : "${build_time:=$(date -u '+%Y-%m-%d %H:%M:%S UTC')}" -: "${GATEWAY_IMAGE:=ghcr.io/nvidia/openshell/gateway:0.0.109}" +: "${GATEWAY_IMAGE:=quay.io/opendatahub/odh-openshell-gateway:v0.0.109-rhaiv.0@sha256:a80b79e514826e8d57ea137749cf18a6e7f3d92e26bfefe005f3a9c4a55b8bdd}" : "${GATEWAY_API_GATEWAY_NAME:=openshell-grpc-gateway}" : "${GATEWAY_API_GATEWAY_NAMESPACE:=openshift-ingress}" @@ -120,7 +120,7 @@ load_cluster_driver() { # shellcheck source=/dev/null source "${path}" local fn - for fn in cluster_up cluster_down cluster_teardown cluster_status component_swap component_revert; do + for fn in cluster_up cluster_down cluster_teardown cluster_status cluster_seed component_swap component_revert; do if ! declare -F "${fn}" >/dev/null; then error "Driver '${driver}' does not implement ${fn}" return 1 @@ -243,7 +243,7 @@ print(docs[0]["id"] if docs else "") } # Restrict a Keycloak client representation to this console origin. -# Spec: oidc-integration Identity Provider Client Security — no wildcards. +# Spec: oidc-integration Identity Provider Client Security: no wildcards. keycloak_client_with_console_redirects() { local console_host="$1" python3 -c 'import json,sys @@ -253,3 +253,18 @@ doc["redirectUris"]=[f"https://{host}/auth/callback", f"https://{host}"] json.dump(doc, sys.stdout) ' "${console_host}" } + +# SKIP_SEED and SEED_STRICT apply to Kind and OpenShift. KIND_* names remain aliases. +skip_seed() { + case "${SKIP_SEED:-${KIND_SKIP_SEED:-}}" in + true|TRUE|1|yes|YES) return 0 ;; + *) return 1 ;; + esac +} + +seed_strict() { + case "${SEED_STRICT:-${KIND_SEED_STRICT:-}}" in + true|TRUE|1|yes|YES) return 0 ;; + *) return 1 ;; + esac +} diff --git a/scripts/cluster/lib_test.sh b/scripts/cluster/lib_test.sh index 432c369f..91d65b7a 100755 --- a/scripts/cluster/lib_test.sh +++ b/scripts/cluster/lib_test.sh @@ -117,6 +117,18 @@ else FAIL=$((FAIL + 1)) echo 'FAIL: default apply_cluster_rbac does not apply overlay ClusterRole/ClusterRoleBinding' fi +if grep -A20 '^cluster_up()' "${SCRIPT_DIR}/drivers/openshift.sh" | grep -q 'skip_seed'; then + PASS=$((PASS + 1)) +else + FAIL=$((FAIL + 1)) + echo 'FAIL: OpenShift cluster_up does not honor SKIP_SEED' +fi +if grep -A15 'Automatic seeding incomplete' "${SCRIPT_DIR}/drivers/openshift.sh" | grep -q 'seed_strict'; then + PASS=$((PASS + 1)) +else + FAIL=$((FAIL + 1)) + echo 'FAIL: OpenShift seeding does not honor SEED_STRICT' +fi if grep -q 'fail_required_cluster_rbac' "${SCRIPT_DIR}/drivers/openshift.sh"; then FAIL=$((FAIL + 1)) echo 'FAIL: OpenShift still fails the whole up when cluster RBAC cannot be applied' @@ -144,7 +156,7 @@ assert_ok "openshift driver loads" bash -c ' source "'"${SCRIPT_DIR}"'/lib.sh" CLUSTER_DRIVER=openshift load_cluster_driver - declare -F cluster_up cluster_down cluster_teardown cluster_status component_swap component_revert >/dev/null + declare -F cluster_up cluster_down cluster_teardown cluster_status cluster_seed component_swap component_revert >/dev/null ' CLUSTER_DRIVER=not-a-driver diff --git a/scripts/cluster/seed.sh b/scripts/cluster/seed.sh new file mode 100755 index 00000000..91523d96 --- /dev/null +++ b/scripts/cluster/seed.sh @@ -0,0 +1,8 @@ +#!/usr/bin/env bash +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# shellcheck source=lib.sh +source "${SCRIPT_DIR}/lib.sh" +load_cluster_driver +cluster_seed diff --git a/scripts/kind/lib.sh b/scripts/kind/lib.sh index 0e945eba..ad5cb927 100755 --- a/scripts/kind/lib.sh +++ b/scripts/kind/lib.sh @@ -44,6 +44,21 @@ fi : "${CPK_LOG:=/tmp/cloud-provider-kind.log}" DNS_CONTAINER_NAME="${KIND_CLUSTER_NAME}-dns" +# SKIP_SEED and SEED_STRICT apply to Kind and OpenShift. KIND_* names remain aliases. +skip_seed() { + case "${SKIP_SEED:-${KIND_SKIP_SEED:-}}" in + true|TRUE|1|yes|YES) return 0 ;; + *) return 1 ;; + esac +} + +seed_strict() { + case "${SEED_STRICT:-${KIND_SEED_STRICT:-}}" in + true|TRUE|1|yes|YES) return 0 ;; + *) return 1 ;; + esac +} + # --- Cluster helpers --- cluster_exists() { diff --git a/scripts/kind/seed.sh b/scripts/kind/seed.sh index f643a9cb..dd0dbd6c 100755 --- a/scripts/kind/seed.sh +++ b/scripts/kind/seed.sh @@ -9,15 +9,16 @@ # once the swapped-in images are live, exercises the branch's own contract. # # Local `make kind-up` invokes this inline by default. CI defers it: it sets -# KIND_SKIP_SEED=true on kind-up and runs `make kind-seed` after the swap. +# SKIP_SEED=true on kind-up and runs `make kind-seed` after the swap. # # Environment: # DATABASE_PROVIDER cnpg | deployment (default: deployment). Must match the # provider kind-up provisioned infrastructure for. -# KIND_SEED_STRICT when "true", a seeding failure exits non-zero instead of +# SEED_STRICT when "true", a seeding failure exits non-zero instead of # only warning. CI sets this so a contract regression fails # the job at the seed step with the real HTTP error, rather # than surfacing later as a confusing discovery failure. +# KIND_SEED_STRICT remains an alias. set -euo pipefail SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" @@ -298,7 +299,7 @@ fi cleanup_pf trap - EXIT echo "" -if [[ -n "${seed_failed}" && "${KIND_SEED_STRICT:-}" == "true" ]]; then - error "Platform seeding failed and KIND_SEED_STRICT=true - failing" +if [[ -n "${seed_failed}" ]] && seed_strict; then + error "Platform seeding failed and SEED_STRICT=true - failing" exit 1 -fi +fi \ No newline at end of file diff --git a/scripts/kind/up.sh b/scripts/kind/up.sh index c3c94bdd..f9c128e2 100755 --- a/scripts/kind/up.sh +++ b/scripts/kind/up.sh @@ -749,12 +749,12 @@ echo "" # --- Seed platform resources via REST API --- # Seeding lives in seed.sh so CI can run it AFTER the component image swap # (see scripts/kind/seed.sh). Local runs seed inline by default; CI sets -# KIND_SKIP_SEED=true here and runs `make kind-seed` once the swapped-in +# SKIP_SEED=true here and runs `make kind-seed` once the swapped-in # working-tree images are live, so the seed exercises the branch's own request # contract instead of the baseline placeholder image kind-up deploys first. -if [[ "${KIND_SKIP_SEED:-}" == "true" ]]; then +if skip_seed; then header "Gateway Provisioning" - info "KIND_SKIP_SEED=true - deferring platform seeding (run 'make kind-seed')" + info "SKIP_SEED=true - deferring platform seeding (run 'make kind-seed')" echo "" else "${SCRIPT_DIR}/seed.sh" diff --git a/specs/platform/e2e-testing.spec.md b/specs/platform/e2e-testing.spec.md index e8d5cc94..781e92af 100644 --- a/specs/platform/e2e-testing.spec.md +++ b/specs/platform/e2e-testing.spec.md @@ -377,8 +377,9 @@ To exercise the periodic path without waiting for production defaults (5m sweep short Go duration strings (for example `30s`; any positive value accepted by `time.ParseDuration` is valid). Immediately after gateway provisioning succeeds, the suite SHALL seed a synthetic orphaned managed namespace (`openshell-e2e-orphan-*`) -labeled with both required management labels (`hypershell.redhat.io/managed=true` -and `app.kubernetes.io/managed-by=hypershell-control-plane`) and a name matching +labeled with the three required ownership labels (`hypershell.redhat.io/managed=true`, +`app.kubernetes.io/managed-by=hypershell-control-plane`, and +`hypershell.redhat.io/instance=`) and a name matching the gateway prefix (not `openshell-db-*`), annotate it with a backdated `hypershell.redhat.io/gc-eligible-since` timestamp so the next sweep can reap without waiting a full grace period. Steps 3–10 SHALL run while the periodic @@ -414,7 +415,7 @@ removal of the in-namespace sandbox resources (see - GIVEN the Kind overlay has shortened `GATEWAY_NAMESPACE_GC_INTERVAL` and `GATEWAY_NAMESPACE_GC_GRACE_PERIOD` (for example `30s`) - AND a synthetic managed namespace was seeded after gateway provisioning with - both management labels, a gateway-style name, and a backdated + this instance's three ownership labels, a gateway-style name, and a backdated `hypershell.redhat.io/gc-eligible-since` annotation, with no live Gateway backing it - AND steps 3–10 have run while the periodic reaper may have deleted it diff --git a/specs/platform/openshell-gateway-namespace-gc.spec.md b/specs/platform/openshell-gateway-namespace-gc.spec.md index 49c1b2bf..8134f058 100644 --- a/specs/platform/openshell-gateway-namespace-gc.spec.md +++ b/specs/platform/openshell-gateway-namespace-gc.spec.md @@ -38,17 +38,26 @@ Namespace creation and provisioning mechanics are defined in - **Gateway namespace** - the Kubernetes namespace a gateway's workloads run in. Its name is API-assigned from the Gateway identifier and is prefixed `openshell-` (e.g. `openshell-a14873d1631f1b74`). -- **Managed namespace** - a namespace the control plane created and is - responsible for, identified by carrying BOTH of the labels the control plane +- **Control-plane instance** - one HyperShell control plane (the + `hypershell-controller` in its platform namespace). Its identity is unique to + that controller: the Kubernetes namespace the controller pod runs in, which is + unique on the cluster. In cluster this is `HYPERSHELL_NAMESPACE`, taken from the + pod's own namespace via the downward API so two controllers cannot accidentally + share a copied static value. Multiple instances MAY share a cluster (for + example stage alongside an e2e run, or two developers' local-dev environments). + Each instance has its own API server and therefore its own set of live Gateways. +- **Managed namespace** - a namespace this control-plane instance created and is + responsible for, identified by carrying ALL of the labels the control plane stamps at creation: - `app.kubernetes.io/managed-by=hypershell-control-plane` - `hypershell.redhat.io/managed=true` - Periodic garbage collection sweeps managed namespaces whose names match the - gateway prefix (`openshell-`) and excludes ManagedDatabase namespaces - (`openshell-db-`). This name-based scope keeps pre-existing orphaned - gateway namespaces eligible for GC without a label migration. A Gateway - pointed at a pre-existing or shared namespace can never cause that namespace to - be reaped. + - `hypershell.redhat.io/instance=` + Periodic garbage collection sweeps only namespaces owned by this instance whose + names match the gateway prefix (`openshell-`) and excludes ManagedDatabase + namespaces (`openshell-db-`). A namespace owned by a different instance, or + lacking this instance's identity label, is never listed, annotated, or reaped. + A Gateway pointed at a pre-existing or shared namespace can never cause that + namespace to be reaped. - **Orphaned namespace** - a gateway namespace (matching the gateway prefix, not the database prefix) for which no live Gateway exists (no Gateway in the API server maps to it). This is the sole trigger for garbage collection. @@ -92,7 +101,11 @@ live outside its namespace, because namespace deletion does not reach them: Namespace deletion SHALL be best-effort and idempotent: an already-absent or already-terminating namespace is treated as success, and a namespace that is not -managed by this control plane SHALL NOT be deleted. +managed by this control-plane instance SHALL NOT be deleted. A namespace that +carries `hypershell.redhat.io/instance` set to a different instance SHALL NOT be +deleted. A legacy namespace that carries the two management labels but no +instance label MAY still be deleted on this path, because the delete is keyed to +a Gateway from this instance's API server rather than a cluster-wide sweep. Namespace deletion SHALL NOT be gated on the number of active sandboxes. A delete is processed and the namespace is removed; if the namespace no longer exists when @@ -120,12 +133,25 @@ the delete is processed, the delete is considered complete. - WHEN the control plane processes the Gateway delete event - THEN it SHALL NOT delete that namespace +#### Scenario: Delete does not touch another instance's namespace + +- GIVEN a Gateway delete event whose namespace carries + `hypershell.redhat.io/instance` set to a different control-plane instance +- WHEN the control plane processes the Gateway delete event +- THEN it SHALL NOT delete that namespace + ### Requirement: Periodic Garbage Collection of Orphaned Namespaces The control plane SHALL run a background reconciler that periodically lists -managed namespaces and reaps gateway workload namespaces (`openshell-`, -excluding ManagedDatabase namespaces `openshell-db-`) that have been orphaned -(no live Gateway) for at least the grace period. The sweep interval defaults to +namespaces owned by this control-plane instance and reaps gateway workload +namespaces (`openshell-`, excluding ManagedDatabase namespaces +`openshell-db-`) that have been orphaned (no live Gateway in this instance's +API server) for at least the grace period. The list selector SHALL include +`hypershell.redhat.io/instance=` in addition to the two +management labels, so a namespace created by another HyperShell instance on the +same cluster is never observed as an orphan of this instance. If +`HYPERSHELL_NAMESPACE` is empty, the reconciler SHALL abort the sweep rather than +list by the generic management labels alone. The sweep interval defaults to 5 minutes and the grace period defaults to 10 minutes. Garbage collection SHALL be enabled by default and configurable without code changes via environment variables: @@ -134,18 +160,35 @@ variables: - `GATEWAY_NAMESPACE_GC_INTERVAL` (default `5m`) - `GATEWAY_NAMESPACE_GC_GRACE_PERIOD` (default `10m`) -Reaping SHALL be best-effort and idempotent, and SHALL only ever delete managed -gateway workload namespaces (matching the gateway prefix, not the database -prefix). +Reaping SHALL be best-effort and idempotent, and SHALL only ever delete gateway +workload namespaces owned by this instance (matching the gateway prefix, not the +database prefix, and carrying this instance's identity label). #### Scenario: Orphaned gateway namespace reaped after grace period -- GIVEN a managed namespace with a gateway-prefixed name (`openshell-`, not - `openshell-db-`) with no live Gateway +- GIVEN a namespace owned by this control-plane instance with a gateway-prefixed + name (`openshell-`, not `openshell-db-`) with no live Gateway in this + instance's API server - AND it has been continuously orphaned for longer than the grace period - WHEN the garbage-collection reconciler sweeps - THEN it SHALL delete the namespace +#### Scenario: Another instance's gateway namespace is not treated as orphaned + +- GIVEN two HyperShell control-plane instances share a cluster +- AND instance A has live Gateways whose namespaces are labeled + `hypershell.redhat.io/instance=` +- WHEN instance B's garbage-collection reconciler sweeps +- THEN it SHALL NOT list, annotate, or delete instance A's namespaces +- AND it SHALL NOT record them as orphaned of instance B + +#### Scenario: Unlabeled legacy namespace is not swept + +- GIVEN a namespace that carries the two management labels but not + `hypershell.redhat.io/instance` +- WHEN this instance's garbage-collection reconciler sweeps +- THEN it SHALL NOT list, annotate, or delete that namespace + #### Scenario: ManagedDatabase namespace is not reaped - GIVEN a managed namespace named `openshell-db-` with no live Gateway @@ -161,12 +204,51 @@ prefix). #### Scenario: Delete event missed during downtime is recovered -- GIVEN the control plane was down when a Gateway was deleted, so the namespace +- GIVEN the control plane was down when a Gateway is deleted, so the namespace was never reaped by the delete path +- AND the namespace carries this instance's identity label - WHEN the control plane restarts and the garbage-collection reconciler sweeps - THEN it SHALL observe the namespace as orphaned and, once the grace period has elapsed, delete it +### Requirement: Stamp This Instance's Identity on Namespaces It Manages + +When the control plane creates or reconciles a namespace it owns, it SHALL stamp +`hypershell.redhat.io/instance` with a value unique to that controller (the +namespace the controller pod runs in, `HYPERSHELL_NAMESPACE`) together with the +two management labels. If the namespace already carries a different instance +identity, the control plane SHALL NOT adopt, relabel, or delete it. Reconciling a +live Gateway SHALL add the instance label to a legacy unlabeled namespace this +instance is actively managing, so a later orphan can be reaped by this instance's +periodic GC. An empty `HYPERSHELL_NAMESPACE` SHALL NOT create or relabel a +managed namespace. Two controllers on the same cluster SHALL NOT share an +instance identity. + +#### Scenario: Created gateway namespace is labeled for this instance + +- GIVEN a Gateway ADDED event for a namespace that does not yet exist +- WHEN the control plane creates the namespace +- THEN the namespace SHALL carry `hypershell.redhat.io/instance` equal to this + controller's unique identity (`HYPERSHELL_NAMESPACE`, the namespace the + controller pod runs in) + +#### Scenario: Live reconcile labels a legacy namespace this instance owns + +- GIVEN a live Gateway whose namespace carries the two management labels but no + instance label +- WHEN the control plane reconciles that Gateway +- THEN it SHALL stamp `hypershell.redhat.io/instance` for this instance +- AND it SHALL NOT change an instance label that already identifies a different + instance + +#### Scenario: Foreign instance namespace is not adopted + +- GIVEN a namespace labeled `hypershell.redhat.io/instance` for a different + control-plane instance +- WHEN this instance would create or reconcile a Gateway in that namespace +- THEN it SHALL NOT overwrite the instance label +- AND it SHALL NOT deploy into or delete that namespace + ### Requirement: Grace Period Prevents Premature Deletion The control plane SHALL NOT delete an orphaned namespace immediately. It SHALL @@ -301,7 +383,10 @@ real-time guarantee. |----------|-----------| | GC triggers on orphaning (no live Gateway), not on `phase` being `Degraded`/`Failed` | A gateway that still exists - even if unhealthy - is the health reconciler's and the operator's concern; only the absence of a backing Gateway unambiguously means the namespace is garbage. This avoids reaping a namespace an operator is still debugging. | | Exclude `openshell-db-*` managed namespaces from periodic GC | ManagedDatabase CNPG namespaces share the management labels but are owned by the ManagedDatabase reconciler; the stable `openshell-db-` prefix distinguishes them without requiring a label migration on existing gateway namespaces. | -| Require BOTH management labels before deleting | Defense in depth: even if a label selector over-returns, a namespace not created by this control plane (e.g. a shared or pre-existing namespace) is never deleted. | +| Require this instance's identity label before periodic GC | Two HyperShell controllers on one cluster share the generic management labels. Without a value unique to that controller (`hypershell.redhat.io/instance=`), instance B's sweep treats instance A's live gateways as orphans (they are absent from B's API server) and would delete them after the grace period. The identity is the controller pod's namespace from the downward API so it cannot be a copied static string. | +| Leave unlabeled legacy namespaces out of periodic GC | Fail closed on coexistence: a namespace without this instance's identity might belong to another HyperShell that predates the label. Delete-driven cleanup of a Gateway from this API server may still reap an unlabeled managed namespace, because that path is keyed to this instance's own Gateway rather than a cluster-wide list. | +| Require BOTH management labels plus this instance's identity before deleting | Defense in depth: even if a label selector over-returns, a namespace not created by this control-plane instance (another HyperShell, a shared namespace, or a pre-existing namespace) is never deleted by periodic GC. | +| Stamp the instance label on create and on live reconcile | Periodic GC can only see namespaces this instance labeled. Stamping at create covers new gateways; stamping on live reconcile migrates this instance's pre-label namespaces so they remain eligible for a later orphan sweep. | | Grace period persisted on the namespace annotation | The delay must survive control-plane restarts; storing `gc-eligible-since` on the namespace makes the timer durable without a separate store. | | Abort the whole sweep if Gateways cannot be listed | An empty or failed Gateway list would make every managed namespace look orphaned; aborting is the only safe response to avoid mass reaping of live namespaces. | | Delete is best-effort and not gated on sandbox count | Deletion is idempotent - process the delete, remove the namespace, and if it is already gone consider the delete done. The sandbox count is a warning surfaced to the operator, not a backend precondition. | diff --git a/specs/platform/openshift-development.spec.md b/specs/platform/openshift-development.spec.md index 67712def..d0011f34 100644 --- a/specs/platform/openshift-development.spec.md +++ b/specs/platform/openshift-development.spec.md @@ -204,7 +204,7 @@ command SHALL still refuse reserved names and namespaces labeled as a different HyperShell environment. The command SHALL wait until each project is gone before it reports success, rather than return after it has only requested deletion. When `oc delete project` is forbidden, the command SHALL delete HyperShell resources inside both projects -— including the bundled Keycloak workload, which is unlabeled — wait for those +(including the bundled Keycloak workload, which is unlabeled), wait for those deletes, and leave the projects. The `make openshift-status` command SHALL report the cluster, the environment namespaces, the pods, the services, the Routes, the Gateway status, and the diff --git a/tests/e2e/e2e-openshell.sh b/tests/e2e/e2e-openshell.sh index 17414955..f1b82c72 100755 --- a/tests/e2e/e2e-openshell.sh +++ b/tests/e2e/e2e-openshell.sh @@ -493,6 +493,7 @@ metadata: labels: hypershell.redhat.io/managed: "true" app.kubernetes.io/managed-by: hypershell-control-plane + hypershell.redhat.io/instance: "${E2E_HS_NAMESPACE}" annotations: hypershell.redhat.io/gc-eligible-since: "${ORPHAN_ELIGIBLE_SINCE}" EOF @@ -1650,8 +1651,9 @@ else # Deleting the Gateway via the API drives the control-plane delete path # (watch-delete-events.spec.md): DeleteGatewayResources then # DeleteManagedNamespace, best-effort and idempotent. The gateway namespace is - # managed (carries both hypershell.redhat.io/managed=true and - # app.kubernetes.io/managed-by=hypershell-control-plane), so it MUST be reaped. + # managed (carries hypershell.redhat.io/managed=true, + # app.kubernetes.io/managed-by=hypershell-control-plane, and + # hypershell.redhat.io/instance=), so it MUST be reaped. # Any namespace missed by the delete path is later swept by the # NamespaceGCReconciler. See openshell-gateway-namespace-gc.spec.md # (HYPERSHELL-96, HYPERSHELL-78). From 3be910a4e6cb2989e12587b24788faf052672dab Mon Sep 17 00:00:00 2001 From: Kyle Squizzato Date: Wed, 2 Sep 2026 12:52:30 -0700 Subject: [PATCH 07/14] fix: reclaim unlabeled gateway namespaces and harden OpenShift local-dev auth Periodic GC backfills hypershell.redhat.io/instance on leftover unlabeled openshell-* namespaces so missed-delete orphans are not invisible. OpenShift JWT now comes from the overlay (API_ENV=development_oidc), and a Forbidden prefixed ClusterRole falls back to binding the existing hypershell-controller. Assisted-by: Cursor Grok 4.6 --- DEVELOPMENT.md | 14 +- Makefile | 3 +- .../internal/gateway/namespace.go | 51 +++++- .../internal/gateway/namespace_test.go | 87 +++++++++++ .../internal/reconciler/namespace.go | 25 ++- .../internal/reconciler/namespace_test.go | 98 +++++++++++- deploy/base/controller-rbac.yaml | 6 +- deploy/openshift/kustomization.yaml | 23 +++ scripts/cluster/drivers/openshift.sh | 147 +++++++++++------- scripts/cluster/lib_test.sh | 45 +++++- scripts/cluster/rewrite-namespaces.py | 4 +- skills/RECONCILE.md | 14 +- specs/platform/e2e-testing.spec.md | 1 + .../openshell-gateway-namespace-gc.spec.md | 75 +++++++-- specs/platform/openshift-development.spec.md | 49 ++++-- 15 files changed, 531 insertions(+), 111 deletions(-) diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md index 21c43d5f..30c00414 100644 --- a/DEVELOPMENT.md +++ b/DEVELOPMENT.md @@ -340,13 +340,16 @@ manifests (with prune scoped to this environment), registers the web-console Route as the Keycloak `hypershell-frontend` redirect URI, seeds a ManagedCluster, GatewayRelease, ManagedDatabase, and Gateway from this machine against the API and Keycloak Routes (the API server image has no `curl`), and -prints the API, web-console, and Keycloak Routes. The gateway base domain is +prints the API, web-console, and Keycloak Routes. The overlay sets +`API_ENV=development_oidc` on the API server so `--enable-jwt=true` is not +clobbered by the default `development` environment. Route-derived OIDC values +(`KC_HOSTNAME`, console redirect URIs, gateway issuer) are applied after the +Routes exist. The gateway base domain is read from the shared Gateway's listener hostname, not from `GATEWAY_API_BASE_DOMAIN`. When ClusterRole create is forbidden, the command -warns and continues. `OPENSHIFT_USE_EXISTING_CLUSTERROLE=true` skips creating a -per-environment ClusterRole and instead creates `${namespace}-dev-hypershell-controller` -ClusterRoleBinding whose `roleRef` is the existing cluster-wide -`hypershell-controller` ClusterRole. It never applies unprefixed +binds `${namespace}-dev-hypershell-controller` to the existing cluster-wide +ClusterRole `hypershell-controller` if that ClusterRole exists. Only if that +fallback also fails does it warn and continue. It never applies unprefixed `hypershell-controller`. `make openshift-down` deletes this environment's `${namespace}-dev-*` ClusterRoles and ClusterRoleBindings and does not delete stage's `hypershell-controller`. @@ -396,7 +399,6 @@ active swaps. | `KIND_NO_SUDO` | (unset) | Set to `true` to skip sudo operations | | `KIND_DNS_PORT` | `5553` | Host port for CoreDNS container | | `OPENSHIFT_NAMESPACE` | `oc project -q` | Override for the platform namespace. Unset, the current oc project is used. Max 54 chars; Keycloak lands in `${name}-keycloak`. | -| `OPENSHIFT_USE_EXISTING_CLUSTERROLE` | (unset) | Set to `true` to bind this environment to the existing ClusterRole `hypershell-controller` instead of creating `${namespace}-dev-hypershell-controller`. | | `SKIP_SEED` | (unset) | Set to `true` to skip ManagedCluster/GatewayRelease/ManagedDatabase/Gateway seeding on `make kind-up` and `make openshift-up`. `KIND_SKIP_SEED` is still accepted. | | `SEED_STRICT` | (unset) | Set to `true` to fail `make kind-up` / `make kind-seed` / `make openshift-up` if seeding is incomplete. `KIND_SEED_STRICT` is still accepted. | | `GATEWAY_API_GATEWAY_NAME` | `openshell-grpc-gateway` | Pre-existing shared Gateway name | diff --git a/Makefile b/Makefile index b0ac7749..5c8b60cb 100644 --- a/Makefile +++ b/Makefile @@ -97,7 +97,6 @@ help: @echo " Kind uses KIND_NAMESPACE (default hypershell-system); kind-up creates the cluster." @echo " OpenShift uses the current oc project (oc project -q); OPENSHIFT_NAMESPACE overrides." @echo " OpenShift requires an existing cluster; openshift-up does not create one." - @echo " OPENSHIFT_USE_EXISTING_CLUSTERROLE=true binds to ClusterRole hypershell-controller instead of creating a per-env copy." @echo "" @echo " -up Deploy the stack (Kind also creates the cluster)" @echo " -down Remove the environment namespace(s)" @@ -320,7 +319,7 @@ export api_server_local control_plane_local web_console_local export build_version build_time export API_HOSTNAME CONSOLE_HOSTNAME HEALTH_HOSTNAME KEYCLOAK_HOSTNAME METRICS_HOSTNAME KEYCLOAK_OIDC_ISSUER export KIND_DNS_PORT -export OPENSHIFT_NAMESPACE OPENSHIFT_IMAGE_REGISTRY OPENSHIFT_USE_EXISTING_CLUSTERROLE SKIP_SEED SEED_STRICT +export OPENSHIFT_NAMESPACE OPENSHIFT_IMAGE_REGISTRY SKIP_SEED SEED_STRICT export GATEWAY_API_GATEWAY_NAME GATEWAY_API_GATEWAY_NAMESPACE GATEWAY_IMAGE # Build cloud-provider-kind from a fork that adds BackendTLSPolicy support diff --git a/components/control-plane/internal/gateway/namespace.go b/components/control-plane/internal/gateway/namespace.go index f465de70..e81e1125 100644 --- a/components/control-plane/internal/gateway/namespace.go +++ b/components/control-plane/internal/gateway/namespace.go @@ -2,6 +2,7 @@ package gateway import ( "context" + "errors" "fmt" "log" "strings" @@ -62,6 +63,52 @@ func ManagedNamespaceSelector(instance string) string { ManagedLabel, ManagedLabelValue, ManagedByLabel, ManagedByValue, InstanceLabel, instance) } +// LegacyUnlabeledSelector selects HyperShell-managed namespaces that predate the +// instance label: both management labels, and no hypershell.redhat.io/instance +// key. Periodic GC uses this to claim leftover gateway namespaces so a +// missed-delete orphan is not invisible forever. Namespaces already labeled for +// any instance are excluded. +func LegacyUnlabeledSelector() string { + return fmt.Sprintf("%s=%s,%s=%s,!%s", + ManagedLabel, ManagedLabelValue, ManagedByLabel, ManagedByValue, InstanceLabel) +} + +// BackfillInstanceLabels stamps this instance's identity onto unlabeled legacy +// gateway namespaces (both management labels, no instance label, openshell-* +// and not openshell-db-*). It never creates namespaces, never overwrites a +// different instance label, and never claims ManagedDatabase namespaces. An +// empty instance is a configuration error: unlabeled namespaces must not be +// claimed without an identity. +func BackfillInstanceLabels(ctx context.Context, client kubernetes.Interface, instance string) error { + if instance == "" { + return fmt.Errorf("instance identity is empty; refusing to backfill namespace labels") + } + list, err := client.CoreV1().Namespaces().List(ctx, metav1.ListOptions{ + LabelSelector: LegacyUnlabeledSelector(), + }) + if err != nil { + return fmt.Errorf("list unlabeled managed namespaces: %w", err) + } + var errs []error + for i := range list.Items { + ns := &list.Items[i] + // Defense in depth: a selector over-return (or a client that ignores + // selectors) must not claim unmanaged or already-labeled namespaces. + if !hasManagementLabels(ns) || ns.Labels[InstanceLabel] != "" { + continue + } + if !isGatewayWorkloadName(ns.Name) { + continue + } + if err := EnsureManagedNamespace(ctx, client, ns.Name, instance); err != nil { + errs = append(errs, fmt.Errorf("claim unlabeled namespace %s: %w", ns.Name, err)) + continue + } + log.Printf("INFO claimed unlabeled legacy namespace %s for instance %s", ns.Name, instance) + } + return errors.Join(errs...) +} + // ManagedNamespaceLabels is the label set stamped on namespaces this instance // creates and reconciles. func ManagedNamespaceLabels(instance string) map[string]string { @@ -153,8 +200,8 @@ func DeleteManagedNamespace(ctx context.Context, client kubernetes.Interface, na // EnsureManagedNamespace creates namespace if it is absent, and otherwise // reconciles the management and instance labels onto it. It refuses to adopt a // namespace already labeled as a different control-plane instance. An empty -// instance is a configuration error: unlabeled namespaces are invisible to -// periodic GC and unsafe to claim on a shared cluster. +// instance is a configuration error: unlabeled namespaces must not be claimed +// without an identity. func EnsureManagedNamespace(ctx context.Context, client kubernetes.Interface, namespace, instance string) error { if instance == "" { return fmt.Errorf("refusing to manage namespace %s without a control-plane instance identity", namespace) diff --git a/components/control-plane/internal/gateway/namespace_test.go b/components/control-plane/internal/gateway/namespace_test.go index 7cbb0d24..e9d6618e 100644 --- a/components/control-plane/internal/gateway/namespace_test.go +++ b/components/control-plane/internal/gateway/namespace_test.go @@ -43,6 +43,14 @@ func TestManagedNamespaceSelector(t *testing.T) { } } +func TestLegacyUnlabeledSelector(t *testing.T) { + got := LegacyUnlabeledSelector() + want := "hypershell.redhat.io/managed=true,app.kubernetes.io/managed-by=hypershell-control-plane,!hypershell.redhat.io/instance" + if got != want { + t.Errorf("LegacyUnlabeledSelector() = %q, want %q", got, want) + } +} + func TestIsManagedNamespace(t *testing.T) { tests := []struct { name string @@ -241,6 +249,85 @@ func TestEnsureManagedNamespace(t *testing.T) { }) } +func TestBackfillInstanceLabels(t *testing.T) { + ctx := context.Background() + + t.Run("stamps unlabeled gateway namespaces", func(t *testing.T) { + unlabeled := managedNamespaceForInstance("openshell-legacy", "", nil) + client := fake.NewSimpleClientset(unlabeled) + if err := BackfillInstanceLabels(ctx, client, "hypershell"); err != nil { + t.Fatalf("BackfillInstanceLabels() error = %v", err) + } + got, err := client.CoreV1().Namespaces().Get(ctx, "openshell-legacy", metav1.GetOptions{}) + if err != nil { + t.Fatalf("get namespace: %v", err) + } + if got.Labels[InstanceLabel] != "hypershell" { + t.Errorf("instance label = %q, want hypershell", got.Labels[InstanceLabel]) + } + }) + + t.Run("does not claim unlabeled ManagedDatabase namespaces", func(t *testing.T) { + db := managedNamespaceForInstance("openshell-db-a1b2c3d4e5f67890", "", nil) + client := fake.NewSimpleClientset(db) + if err := BackfillInstanceLabels(ctx, client, "hypershell"); err != nil { + t.Fatalf("BackfillInstanceLabels() error = %v", err) + } + got, err := client.CoreV1().Namespaces().Get(ctx, "openshell-db-a1b2c3d4e5f67890", metav1.GetOptions{}) + if err != nil { + t.Fatalf("get namespace: %v", err) + } + if _, ok := got.Labels[InstanceLabel]; ok { + t.Errorf("ManagedDatabase namespace was claimed, want unlabeled") + } + }) + + t.Run("does not overwrite a foreign instance label", func(t *testing.T) { + foreign := managedNamespaceForInstance("openshell-stage", "hypershell-stage", nil) + client := fake.NewSimpleClientset(foreign) + if err := BackfillInstanceLabels(ctx, client, "hypershell"); err != nil { + t.Fatalf("BackfillInstanceLabels() error = %v", err) + } + got, err := client.CoreV1().Namespaces().Get(ctx, "openshell-stage", metav1.GetOptions{}) + if err != nil { + t.Fatalf("get namespace: %v", err) + } + if got.Labels[InstanceLabel] != "hypershell-stage" { + t.Errorf("instance label = %q, want hypershell-stage", got.Labels[InstanceLabel]) + } + }) + + t.Run("refuses an empty instance identity", func(t *testing.T) { + unlabeled := managedNamespaceForInstance("openshell-legacy", "", nil) + client := fake.NewSimpleClientset(unlabeled) + if err := BackfillInstanceLabels(ctx, client, ""); err == nil { + t.Fatalf("BackfillInstanceLabels() error = nil, want empty instance error") + } + got, err := client.CoreV1().Namespaces().Get(ctx, "openshell-legacy", metav1.GetOptions{}) + if err != nil { + t.Fatalf("get namespace: %v", err) + } + if _, ok := got.Labels[InstanceLabel]; ok { + t.Errorf("empty instance claimed a namespace, want unlabeled") + } + }) + + t.Run("does not claim an unmanaged openshell namespace", func(t *testing.T) { + unmanaged := &corev1.Namespace{ObjectMeta: metav1.ObjectMeta{Name: "openshell-shared"}} + client := fake.NewSimpleClientset(unmanaged) + if err := BackfillInstanceLabels(ctx, client, "hypershell"); err != nil { + t.Fatalf("BackfillInstanceLabels() error = %v", err) + } + got, err := client.CoreV1().Namespaces().Get(ctx, "openshell-shared", metav1.GetOptions{}) + if err != nil { + t.Fatalf("get namespace: %v", err) + } + if len(got.Labels) != 0 { + t.Errorf("unmanaged namespace was labeled %v, want untouched", got.Labels) + } + }) +} + func TestMarkGCEligible(t *testing.T) { ctx := context.Background() now := time.Date(2026, 8, 17, 12, 0, 0, 0, time.UTC) diff --git a/components/control-plane/internal/reconciler/namespace.go b/components/control-plane/internal/reconciler/namespace.go index 5b3ec843..28e28597 100644 --- a/components/control-plane/internal/reconciler/namespace.go +++ b/components/control-plane/internal/reconciler/namespace.go @@ -45,10 +45,14 @@ const ( // this control-plane instance created but that no longer have a live Gateway in // this instance's API server. Other HyperShell instances on the same cluster are // ignored: the sweep selects on hypershell.redhat.io/instance=. -// This reaps namespaces orphaned by a delete event missed while the control -// plane was down, and namespaces whose gateway failed to bootstrap and was then -// deleted. Reaping is best-effort and idempotent, and is delayed by a grace -// period recorded on the namespace itself so it survives restarts. +// At the start of each sweep it claims unlabeled legacy gateway namespaces +// (both management labels, no instance label) by stamping this instance, so a +// missed-delete orphan from before the instance label existed is not left +// invisible to GC. This reaps namespaces orphaned by a delete event missed +// while the control plane was down, and namespaces whose gateway failed to +// bootstrap and was then deleted. Reaping is best-effort and idempotent, and is +// delayed by a grace period recorded on the namespace itself so it survives +// restarts. // // See openshell-gateway-namespace-gc.spec.md (HYPERSHELL-78). type NamespaceGCReconciler struct { @@ -132,6 +136,19 @@ func (r *NamespaceGCReconciler) reconcileOnce(ctx context.Context) { return } + // Claim pre-instance-label leftover gateway namespaces so a missed-delete + // orphan (two management labels, no instance label, no live Gateway) is + // visible to the instance-scoped sweep below. Live unlabeled namespaces + // are also stamped here; the live set then keeps them from being orphaned. + // Failure to claim must not abort the labeled sweep: already-labeled + // orphans can still be reaped. Foreign instance labels are never overwritten. + backfillCtx, backfillCancel := context.WithTimeout(ctx, namespaceListTimeout) + if err := gateway.BackfillInstanceLabels(backfillCtx, r.client, r.cpNamespace); err != nil { + tickErr = errors.Join(tickErr, err) + log.Printf("WARN namespace gc: backfill unlabeled instance labels: %v", err) + } + backfillCancel() + listCtx, cancel := context.WithTimeout(ctx, namespaceListTimeout) namespaces, err := r.client.CoreV1().Namespaces().List(listCtx, metav1.ListOptions{ LabelSelector: gateway.ManagedNamespaceSelector(r.cpNamespace), diff --git a/components/control-plane/internal/reconciler/namespace_test.go b/components/control-plane/internal/reconciler/namespace_test.go index df3fa386..9ef9d3d6 100644 --- a/components/control-plane/internal/reconciler/namespace_test.go +++ b/components/control-plane/internal/reconciler/namespace_test.go @@ -325,14 +325,96 @@ func TestReconcileOnce_OnlySweepsThisInstance(t *testing.T) { t.Errorf("this instance's orphan was not stamped gc-eligible-since") } - for _, name := range []string{"openshell-stage", "openshell-legacy"} { - got, err := client.CoreV1().Namespaces().Get(ctx, name, metav1.GetOptions{}) - if err != nil { - t.Fatalf("get %s: %v", name, err) - } - if _, ok := got.Annotations[gateway.GCEligibleSinceAnnotation]; ok { - t.Errorf("%s was treated as an orphan of this instance, want ignored", name) - } + legacy, err := client.CoreV1().Namespaces().Get(ctx, "openshell-legacy", metav1.GetOptions{}) + if err != nil { + t.Fatalf("get unlabeled legacy namespace: %v", err) + } + if legacy.Labels[gateway.InstanceLabel] != "hypershell" { + t.Errorf("unlabeled legacy namespace instance = %q, want hypershell after backfill", legacy.Labels[gateway.InstanceLabel]) + } + if legacy.Annotations[gateway.GCEligibleSinceAnnotation] != now.Format(time.RFC3339) { + t.Errorf("unlabeled legacy orphan was not stamped gc-eligible-since after backfill") + } + + foreignUpdated, err := client.CoreV1().Namespaces().Get(ctx, "openshell-stage", metav1.GetOptions{}) + if err != nil { + t.Fatalf("get foreign namespace: %v", err) + } + if foreignUpdated.Labels[gateway.InstanceLabel] != "hypershell-stage" { + t.Errorf("foreign instance label overwritten to %q", foreignUpdated.Labels[gateway.InstanceLabel]) + } + if _, ok := foreignUpdated.Annotations[gateway.GCEligibleSinceAnnotation]; ok { + t.Errorf("foreign instance namespace was treated as an orphan of this instance, want ignored") + } +} + +func TestReconcileOnce_BackfillDoesNotOrphanLiveUnlabeled(t *testing.T) { + ctx := context.Background() + now := time.Date(2026, 8, 17, 12, 0, 0, 0, time.UTC) + unlabeled := nsWithLabels("openshell-legacy", map[string]string{ + gateway.ManagedByLabel: gateway.ManagedByValue, + gateway.ManagedLabel: gateway.ManagedLabelValue, + }, nil) + client := fake.NewSimpleClientset(unlabeled) + r := newTestGC(client, now) + r.liveNamespaces = func(context.Context) (map[string]struct{}, error) { + return map[string]struct{}{"openshell-legacy": {}}, nil + } + + r.reconcileOnce(ctx) + + got, err := client.CoreV1().Namespaces().Get(ctx, "openshell-legacy", metav1.GetOptions{}) + if err != nil { + t.Fatalf("get namespace: %v", err) + } + if got.Labels[gateway.InstanceLabel] != "hypershell" { + t.Errorf("live unlabeled namespace instance = %q, want hypershell after backfill", got.Labels[gateway.InstanceLabel]) + } + if _, ok := got.Annotations[gateway.GCEligibleSinceAnnotation]; ok { + t.Errorf("live unlabeled namespace was marked orphaned, want retained") + } +} + +func TestReconcileOnce_BackfillDoesNotClaimUnlabeledDatabase(t *testing.T) { + ctx := context.Background() + now := time.Date(2026, 8, 17, 12, 0, 0, 0, time.UTC) + db := nsWithLabels("openshell-db-a1b2c3d4e5f67890", map[string]string{ + gateway.ManagedByLabel: gateway.ManagedByValue, + gateway.ManagedLabel: gateway.ManagedLabelValue, + }, nil) + client := fake.NewSimpleClientset(db) + r := newTestGC(client, now) + + r.reconcileOnce(ctx) + + got, err := client.CoreV1().Namespaces().Get(ctx, "openshell-db-a1b2c3d4e5f67890", metav1.GetOptions{}) + if err != nil { + t.Fatalf("get namespace: %v", err) + } + if _, ok := got.Labels[gateway.InstanceLabel]; ok { + t.Errorf("unlabeled ManagedDatabase namespace was claimed, want unlabeled") + } + if _, ok := got.Annotations[gateway.GCEligibleSinceAnnotation]; ok { + t.Errorf("ManagedDatabase namespace was treated as an orphan, want ignored") + } +} + +func TestReconcileOnce_UnlabeledOrphanPastGraceIsReaped(t *testing.T) { + ctx := context.Background() + now := time.Date(2026, 8, 17, 12, 0, 0, 0, time.UTC) + unlabeled := nsWithLabels("openshell-legacy", map[string]string{ + gateway.ManagedByLabel: gateway.ManagedByValue, + gateway.ManagedLabel: gateway.ManagedLabelValue, + }, map[string]string{ + gateway.GCEligibleSinceAnnotation: now.Add(-20 * time.Minute).Format(time.RFC3339), + }) + client := fake.NewSimpleClientset(unlabeled) + r := newTestGC(client, now) + + r.reconcileOnce(ctx) + + if nsExists(t, client, "openshell-legacy") { + t.Fatalf("unlabeled legacy orphan past grace retained, want reaped after backfill") } } diff --git a/deploy/base/controller-rbac.yaml b/deploy/base/controller-rbac.yaml index 698a9ae6..bb47ada6 100644 --- a/deploy/base/controller-rbac.yaml +++ b/deploy/base/controller-rbac.yaml @@ -17,7 +17,11 @@ rules: verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] - apiGroups: ["rbac.authorization.k8s.io"] resources: ["roles", "rolebindings", "clusterroles", "clusterrolebindings"] - verbs: ["bind", "get", "list", "watch", "create", "update", "patch", "delete"] + verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] + - apiGroups: ["rbac.authorization.k8s.io"] + resources: ["clusterroles"] + resourceNames: ["system:openshift:scc:privileged"] + verbs: ["bind"] - apiGroups: ["networking.k8s.io"] resources: ["networkpolicies"] verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] diff --git a/deploy/openshift/kustomization.yaml b/deploy/openshift/kustomization.yaml index 578f1798..0ebebfe6 100644 --- a/deploy/openshift/kustomization.yaml +++ b/deploy/openshift/kustomization.yaml @@ -76,6 +76,24 @@ patches: - op: add path: /spec/template/spec/containers/0/env value: + # API_ENV selects the rh-trex runtime environment. The default + # "development" hard-forces Auth.EnableJWT=false in OverrideConfig() + # AFTER flag parsing, which silently clobbers --enable-jwt=true below + # and leaves REST with no identity so RBAC 401s every request. + # "development_oidc" is the purpose-built environment for a + # self-contained Keycloak: it leaves EnableJWT to the flags and keeps + # the OCM/AMS client mocked (enable-mock=true default), so the + # api-server does NOT require sso.redhat.com secret files. Any overlay + # that enables JWT against a self-hosted IdP MUST set this. Mirrors the + # deploy/kind overlay and the gitops jwt-enforce component. + - name: API_ENV + value: "development_oidc" + # Re-declare the base env: this JSON6902 add REPLACES the whole env + # array, so the base values would otherwise be dropped. + - name: HYPERSHELL_SERVICE_ACCOUNT_PROVISIONER_ADDR + value: "hypershell-controller:9443" + - name: DB_SSLMODE + value: require - name: JWK_CERT_URL valueFrom: secretKeyRef: @@ -91,6 +109,11 @@ patches: - op: add path: /spec/template/spec/containers/0/command/- value: "--enable-authz=true" + # NOTE: the OCM/Red Hat AMS client stays mocked via API_ENV=development_oidc + # (enable-mock=true default), so no explicit --enable-mock flag is needed. + # A self-contained Keycloak deployment has no sso.redhat.com secret files, + # and the AMS client is unused here; JWT auth (enable-jwt) and gateway RBAC + # (RBAC_ENFORCE) are independent of the mock and stay on. - op: add path: /spec/template/spec/containers/0/command/- value: "--jwk-cert-url=$(JWK_CERT_URL)" diff --git a/scripts/cluster/drivers/openshift.sh b/scripts/cluster/drivers/openshift.sh index a50cc677..efbf3369 100755 --- a/scripts/cluster/drivers/openshift.sh +++ b/scripts/cluster/drivers/openshift.sh @@ -266,28 +266,89 @@ check_infrastructure() { info "Gateway base domain: ${GATEWAY_API_BASE_DOMAIN} (from ${gw_ns}/${gw_name} listener)" } -use_existing_clusterrole() { - case "${OPENSHIFT_USE_EXISTING_CLUSTERROLE:-}" in - true|TRUE|1|yes|YES) return 0 ;; - *) return 1 ;; - esac -} - warn_skipped_cluster_rbac() { local err="$1" OPENSHIFT_CLUSTER_RBAC_APPLIED=false warn "Could not apply cluster-scoped RBAC for this environment." warn "${err}" - if use_existing_clusterrole; then - warn "OPENSHIFT_USE_EXISTING_CLUSTERROLE requires ClusterRole hypershell-controller and permission to create a prefixed ClusterRoleBinding to it." - else - warn "Expected this environment's ClusterRole and ClusterRoleBinding:" - warn " ${OPENSHIFT_NAMESPACE}-dev-hypershell-controller" - fi + warn "Expected ClusterRoleBinding ${OPENSHIFT_NAMESPACE}-dev-hypershell-controller" + warn "bound to this environment's ClusterRole or to ClusterRole hypershell-controller." warn "Gateways and sandboxes will not provision until this environment's controller is bound." warn "Do not apply unprefixed ClusterRoleBinding hypershell-controller; that belongs to stage." } +# roleRef is immutable. Delete a leftover ClusterRoleBinding so the next apply +# can point at the intended ClusterRole (prefixed copy vs cluster-wide). +replace_clusterrolebinding_if_role_ref_differs() { + local name="$1" + local want_ref="$2" + local current + current="$(oc_cli get clusterrolebinding "${name}" -o jsonpath='{.roleRef.name}' 2>/dev/null || true)" + if [[ -z "${current}" || "${current}" == "${want_ref}" ]]; then + return 0 + fi + info "Replacing ClusterRoleBinding ${name} (roleRef ${current} -> ${want_ref}; roleRef is immutable)" + oc_cli delete clusterrolebinding "${name}" +} + +apply_rendered_cluster_rbac() { + local rendered err + if ! rendered="$(render_openshift_manifests "$@")"; then + return 1 + fi + assert_expected_cluster_scoped "${rendered}" + if ! err="$(oc_cli apply -f "${rendered}" 2>&1)"; then + rm -f "${rendered}" + LAST_CLUSTER_RBAC_ERR="${err}" + return 1 + fi + printf '%s\n' "${err}" + rm -f "${rendered}" + return 0 +} + +bind_existing_clusterrole() { + local prefix="${OPENSHIFT_NAMESPACE}-dev-" + local crb="${prefix}hypershell-controller" + if ! oc_cli get clusterrole hypershell-controller >/dev/null 2>&1; then + LAST_CLUSTER_RBAC_ERR="ClusterRole hypershell-controller was not found" + return 1 + fi + info "Binding this environment to ClusterRole hypershell-controller (${crb})" + if ! replace_clusterrolebinding_if_role_ref_differs "${crb}" "hypershell-controller"; then + LAST_CLUSTER_RBAC_ERR="Could not replace ClusterRoleBinding ${crb} so roleRef can change" + return 1 + fi + apply_rendered_cluster_rbac \ + --only-namespace __cluster__ \ + --include-cluster-scoped \ + --omit-kinds ClusterRole \ + --keep-role-refs \ + --omit-names "${prefix}hypershell-controller-scc-bind" +} + +apply_prefixed_cluster_rbac() { + local prefix="${OPENSHIFT_NAMESPACE}-dev-" + local crb="${prefix}hypershell-controller" + # Apply the ClusterRole first. Applying it with the ClusterRoleBinding in one + # shot can create a CRB that points at a ClusterRole that never exists, and + # roleRef cannot be patched afterward. + if ! apply_rendered_cluster_rbac \ + --only-namespace __cluster__ \ + --include-cluster-scoped \ + --only-kinds ClusterRole; then + return 1 + fi + if ! replace_clusterrolebinding_if_role_ref_differs "${crb}" "${crb}"; then + LAST_CLUSTER_RBAC_ERR="Could not replace ClusterRoleBinding ${crb} so roleRef can change" + return 1 + fi + apply_rendered_cluster_rbac \ + --only-namespace __cluster__ \ + --include-cluster-scoped \ + --only-kinds ClusterRoleBinding +} + assert_expected_cluster_scoped() { local rendered="$1" local prefix="${OPENSHIFT_NAMESPACE}-dev-" @@ -345,49 +406,35 @@ EOF apply_cluster_rbac() { # Default: apply the overlay ClusterRole and ClusterRoleBinding, names prefixed - # so they never replace stage's hypershell-controller. Workaround: bind this + # so they never replace stage's hypershell-controller. If ClusterRole create is + # Forbidden (escalation prevention: rules the user does not hold), bind this # environment's service account to the existing cluster-wide ClusterRole. local prefix="${OPENSHIFT_NAMESPACE}-dev-" - local render_args=( - --only-namespace __cluster__ - --include-cluster-scoped - ) - local rendered err OPENSHIFT_CLUSTER_RBAC_APPLIED=true + LAST_CLUSTER_RBAC_ERR="" - if use_existing_clusterrole; then - info "OPENSHIFT_USE_EXISTING_CLUSTERROLE: looking up ClusterRole hypershell-controller" - if ! oc_cli get clusterrole hypershell-controller >/dev/null 2>&1; then - error "OPENSHIFT_USE_EXISTING_CLUSTERROLE is set but ClusterRole hypershell-controller was not found." - exit 1 - fi - info "Binding this environment to ClusterRole hypershell-controller (${prefix}hypershell-controller)" - render_args+=( - --omit-kinds ClusterRole - --keep-role-refs - --omit-names "${prefix}hypershell-controller-scc-bind" - ) - else - info "Applying cluster-scoped RBAC from deploy/openshift (${prefix}*)..." + info "Applying cluster-scoped RBAC from deploy/openshift (${prefix}*)..." + if apply_prefixed_cluster_rbac; then + apply_sandbox_scc + return 0 fi - if ! rendered="$(render_openshift_manifests "${render_args[@]}")"; then - exit 1 - fi - assert_expected_cluster_scoped "${rendered}" - if ! err="$(oc_cli apply -f "${rendered}" 2>&1)"; then - rm -f "${rendered}" - if grep -qiE 'forbidden|not found' <<<"${err}"; then - warn_skipped_cluster_rbac "${err}" - apply_sandbox_scc || true - return 0 - fi + local err="${LAST_CLUSTER_RBAC_ERR}" + if ! grep -qi 'forbidden' <<<"${err}"; then error "${err}" return 1 fi - printf '%s\n' "${err}" - rm -f "${rendered}" - apply_sandbox_scc + + warn "Could not create ClusterRole ${prefix}hypershell-controller." + warn "${err}" + warn "Falling back to existing ClusterRole hypershell-controller." + if bind_existing_clusterrole; then + success "Bound this environment to ClusterRole hypershell-controller" + apply_sandbox_scc + return 0 + fi + warn_skipped_cluster_rbac "${LAST_CLUSTER_RBAC_ERR:-${err}}" + apply_sandbox_scc || true } create_bootstrap_secrets() { @@ -667,10 +714,6 @@ configure_oidc_from_routes() { oc_cli set env deployment/keycloak -n "${OPENSHIFT_KEYCLOAK_NAMESPACE}" \ "KC_HOSTNAME=${OPENSHIFT_KC_HOSTNAME}" >/dev/null - info "Configuring API server JWT environment" - oc_cli set env deployment/hypershell-api-server -n "${OPENSHIFT_NAMESPACE}" -c api-server \ - API_ENV=development_oidc >/dev/null - info "Configuring web console OIDC" oc_cli set env deployment/hypershell-web-console -n "${OPENSHIFT_NAMESPACE}" -c web-console \ "OIDC_ISSUER=${OPENSHIFT_OIDC_ISSUER}" \ @@ -974,7 +1017,7 @@ print_banner() { if [[ "${OPENSHIFT_CLUSTER_RBAC_APPLIED:-true}" != "true" ]]; then echo "" warn "This environment's controller is not bound to a ClusterRole." - warn "Gateways/sandboxes need ${OPENSHIFT_NAMESPACE}-dev-hypershell-controller, or OPENSHIFT_USE_EXISTING_CLUSTERROLE=true plus ClusterRole hypershell-controller." + warn "Gateways/sandboxes need ClusterRoleBinding ${OPENSHIFT_NAMESPACE}-dev-hypershell-controller." fi } diff --git a/scripts/cluster/lib_test.sh b/scripts/cluster/lib_test.sh index 91d65b7a..fb9a2418 100755 --- a/scripts/cluster/lib_test.sh +++ b/scripts/cluster/lib_test.sh @@ -103,13 +103,18 @@ if grep -E 'delete clusterrole(binding)? "hypershell-controller' "${SCRIPT_DIR}/ else PASS=$((PASS + 1)) fi -if grep -q 'OPENSHIFT_USE_EXISTING_CLUSTERROLE' "${SCRIPT_DIR}/drivers/openshift.sh" \ - && grep -A50 '^apply_cluster_rbac()' "${SCRIPT_DIR}/drivers/openshift.sh" | grep -q 'get clusterrole hypershell-controller' \ - && grep -A50 '^apply_cluster_rbac()' "${SCRIPT_DIR}/drivers/openshift.sh" | grep -q 'keep-role-refs'; then +if grep -A40 '^bind_existing_clusterrole()' "${SCRIPT_DIR}/drivers/openshift.sh" | grep -q 'get clusterrole hypershell-controller' \ + && grep -A40 '^bind_existing_clusterrole()' "${SCRIPT_DIR}/drivers/openshift.sh" | grep -q 'keep-role-refs'; then PASS=$((PASS + 1)) else FAIL=$((FAIL + 1)) - echo 'FAIL: OPENSHIFT_USE_EXISTING_CLUSTERROLE does not look up ClusterRole hypershell-controller' + echo 'FAIL: bind_existing_clusterrole does not look up ClusterRole hypershell-controller' +fi +if grep -q 'OPENSHIFT_USE_EXISTING_CLUSTERROLE' "${SCRIPT_DIR}/drivers/openshift.sh" "${REPO_ROOT}/Makefile"; then + FAIL=$((FAIL + 1)) + echo 'FAIL: OPENSHIFT_USE_EXISTING_CLUSTERROLE is still present; fallback bind replaced it' +else + PASS=$((PASS + 1)) fi if grep -A50 '^apply_cluster_rbac()' "${SCRIPT_DIR}/drivers/openshift.sh" | grep -q 'Applying cluster-scoped RBAC from deploy/openshift'; then PASS=$((PASS + 1)) @@ -117,6 +122,19 @@ else FAIL=$((FAIL + 1)) echo 'FAIL: default apply_cluster_rbac does not apply overlay ClusterRole/ClusterRoleBinding' fi +if grep -A80 '^apply_cluster_rbac()' "${SCRIPT_DIR}/drivers/openshift.sh" | grep -q 'Falling back to existing ClusterRole hypershell-controller' \ + && grep -A80 '^apply_cluster_rbac()' "${SCRIPT_DIR}/drivers/openshift.sh" | grep -q 'bind_existing_clusterrole'; then + PASS=$((PASS + 1)) +else + FAIL=$((FAIL + 1)) + echo 'FAIL: apply_cluster_rbac does not fall back to ClusterRole hypershell-controller' +fi +if grep -A20 '^replace_clusterrolebinding_if_role_ref_differs()' "${SCRIPT_DIR}/drivers/openshift.sh" | grep -q 'roleRef is immutable'; then + PASS=$((PASS + 1)) +else + FAIL=$((FAIL + 1)) + echo 'FAIL: leftover ClusterRoleBinding roleRef is not replaced before fallback bind' +fi if grep -A20 '^cluster_up()' "${SCRIPT_DIR}/drivers/openshift.sh" | grep -q 'skip_seed'; then PASS=$((PASS + 1)) else @@ -649,6 +667,18 @@ if bad: FAIL=$((FAIL + 1)) echo 'FAIL: namespaced SCC RoleBinding lost built-in system: roleRef' fi + if grep -A1 'name: API_ENV' "${os_out}" | grep -q 'development_oidc'; then + PASS=$((PASS + 1)) + else + FAIL=$((FAIL + 1)) + echo 'FAIL: OpenShift overlay does not set API_ENV=development_oidc' + fi + if grep -q 'name: HYPERSHELL_SERVICE_ACCOUNT_PROVISIONER_ADDR' "${os_out}"; then + PASS=$((PASS + 1)) + else + FAIL=$((FAIL + 1)) + echo 'FAIL: OpenShift overlay dropped HYPERSHELL_SERVICE_ACCOUNT_PROVISIONER_ADDR' + fi else FAIL=$((FAIL + 1)) echo 'FAIL: rewritten overlay render' @@ -658,5 +688,12 @@ else warn "kustomize not installed; skipping overlay render checks" fi +if grep -q 'API_ENV=development_oidc' "${SCRIPT_DIR}/drivers/openshift.sh"; then + FAIL=$((FAIL + 1)) + echo 'FAIL: OpenShift driver still sets API_ENV imperatively; it belongs in deploy/openshift' +else + PASS=$((PASS + 1)) +fi + printf 'OpenShift lifecycle tests: %d passed, %d failed\n' "${PASS}" "${FAIL}" [[ "${FAIL}" -eq 0 ]] diff --git a/scripts/cluster/rewrite-namespaces.py b/scripts/cluster/rewrite-namespaces.py index 1d7882f9..0d31cff1 100755 --- a/scripts/cluster/rewrite-namespaces.py +++ b/scripts/cluster/rewrite-namespaces.py @@ -8,8 +8,8 @@ those ClusterRoles) are prefixed with ${platform}-dev- so each openshift-up environment has its own copy and does not patch stage's hypershell-controller. Built-in ClusterRoles (system:*) are not renamed. --keep-role-refs leaves -roleRef pointing at the existing cluster-wide ClusterRole (workaround for -OPENSHIFT_USE_EXISTING_CLUSTERROLE). +roleRef pointing at the existing cluster-wide ClusterRole when a prefixed +ClusterRole cannot be created. """ from __future__ import annotations diff --git a/skills/RECONCILE.md b/skills/RECONCILE.md index 8249c9db..780ded52 100644 --- a/skills/RECONCILE.md +++ b/skills/RECONCILE.md @@ -133,10 +133,10 @@ Layer 7: web-console/architecture (depends on data-model, security, UI | OS-7 | E2E Script Consolidation | Missing | Intentionally deferred: not local-dev lifecycle | `components/pr-test/` | Future | | OS-8 | Ephemeral CI Environment Provisioning | Missing | Intentionally deferred: not local-dev lifecycle | - | Future | | OS-9 | Environment Access Handoff | Missing | Intentionally deferred: CI-only | - | Future | -| OS-10 | Blessed OpenShift Overlay | Partial | Namespace parameterization, Routes, SCC RoleBindings; gateway base domain discovered from the shared Gateway listener (not `GATEWAY_API_BASE_DOMAIN`). Drift-check CI job deferred. | `deploy/openshift/`, `rewrite-namespaces.py` | OS-W2 | +| OS-10 | Blessed OpenShift Overlay | Partial | Namespace parameterization, Routes, SCC RoleBindings; `API_ENV=development_oidc` in the overlay (not `oc set env`); gateway base domain discovered from the shared Gateway listener (not `GATEWAY_API_BASE_DOMAIN`). Drift-check CI job deferred. | `deploy/openshift/`, `rewrite-namespaces.py` | OS-W2 | | OS-11 | OpenShift CI Workflow Shape | Missing | Intentionally deferred: not local-dev lifecycle | - | Future | | OS-12 | Cluster Infrastructure Prerequisites | Present | `make openshift-up` fails fast when the shared Gateway is missing or not Programmed. GatewayClass is cluster-scoped and not GET-checked (developers typically cannot read it). | `drivers/openshift.sh` `check_infrastructure` | OS-W2 | -| OS-13 | Cluster-Scoped Permissions + SCC/RBAC posture | Present | Default applies prefixed overlay ClusterRole+ClusterRoleBinding (`bind` on clusterroles, same shape as stage). `OPENSHIFT_USE_EXISTING_CLUSTERROLE=true` looks up ClusterRole `hypershell-controller` and binds a prefixed CRB to it. Never touches unprefixed `hypershell-controller`. Down deletes this env's prefixed ClusterRole/CRB. | `rewrite-namespaces.py`, `drivers/openshift.sh` `apply_cluster_rbac`, `deploy/base/controller-rbac.yaml` | OS-W2 | +| OS-13 | Cluster-Scoped Permissions + SCC/RBAC posture | Present | Default applies prefixed overlay ClusterRole then ClusterRoleBinding. If ClusterRole create is Forbidden, bind the prefixed CRB to existing ClusterRole `hypershell-controller` (replace immutable roleRef if needed). Never touches unprefixed `hypershell-controller`. Down deletes this env's prefixed ClusterRole/CRB. | `rewrite-namespaces.py`, `drivers/openshift.sh` `apply_cluster_rbac`, `deploy/base/controller-rbac.yaml` | OS-W2 | Local-dev lifecycle (`make openshift-up` / `down` / component swaps) is implemented. E2E driver completion beyond the OS-W1 manual slice, legacy `pr-test` consolidation, ephemeral CI, access handoff, overlay drift CI, and the OpenShift e2e workflow remain out of scope for this wave. @@ -420,7 +420,7 @@ The OpenShift e2e driver (`tests/e2e/drivers/openshift.sh`) remains a gap for HY | # | Requirement | Status | Gap | Code Location | Wave | |---|-------------|--------|-----|---------------|------| | NGC-1 | Gateway deletion reaps the gateway namespace (cascade + out-of-namespace cleanup) | Present | Delete event deletes the managed namespace (cascading in-namespace resources incl. sandbox pods); ClusterRoleBinding, Keycloak client, cross-namespace credential RBAC cleaned explicitly; best-effort/idempotent; never gated on sandbox count | `gateway/reconciler.go` `DeleteGatewayResources()`, `gateway/namespace.go` | NGC ✅ | -| NGC-2 | Periodic GC of orphaned namespaces (env-configurable) | Present | `NamespaceGCReconciler` sweeps managed namespaces (both management labels required); `GATEWAY_NAMESPACE_GC_ENABLED`/`_INTERVAL`/`_GRACE_PERIOD` default true/5m/10m | `reconciler/namespace.go`, `config/config.go` | NGC ✅ | +| NGC-2 | Periodic GC of orphaned namespaces (env-configurable) | Present | `NamespaceGCReconciler` sweeps instance-labeled managed namespaces; each sweep backfills `hypershell.redhat.io/instance` onto unlabeled legacy gateway namespaces (`openshell-*`, not `openshell-db-*`) then reaps orphans past grace. `GATEWAY_NAMESPACE_GC_ENABLED`/`_INTERVAL`/`_GRACE_PERIOD` default true/5m/10m | `reconciler/namespace.go`, `gateway/namespace.go` `BackfillInstanceLabels`, `config/config.go` | NGC ✅ | | NGC-3 | Grace period prevents premature deletion (durable annotation) | Present | `hypershell.redhat.io/gc-eligible-since` (RFC3339) stamped on and measured from the namespace so it survives restarts; cleared when a live Gateway reappears | `gateway/namespace.go` `MarkGCEligible`/`ClearGCEligible` | NGC ✅ | | NGC-4 | Do not reap namespaces of live gateways (abort on list failure) | Present | Liveness derived from API-reported Gateways; sweep aborts entirely if Gateways cannot be listed; existing gateway preserved regardless of phase (Degraded/Failed) | `reconciler/namespace.go` | NGC ✅ | | NGC-5 | Preserve a durable record before deletion | Present | `GarbageCollected` Event recorded in the control-plane namespace summarizing orphan duration, pod state, and active sandbox count; summary best-effort, never blocks the reap | `reconciler/namespace.go:203` | NGC ✅ | @@ -665,8 +665,12 @@ Added `database/sql` + `lib/pq` to control plane. `rotateDatabaseCredentials()` **Namespace GC (NGC):** `NamespaceGCReconciler` sweeps managed namespaces (both `app.kubernetes.io/managed-by=hypershell-control-plane` and -`hypershell.redhat.io/managed=true` required) and reaps those orphaned past the -grace period. Grace timer persisted on the `hypershell.redhat.io/gc-eligible-since` +`hypershell.redhat.io/managed=true` required, plus +`hypershell.redhat.io/instance=`) and reaps those orphaned +past the grace period. Each sweep first claims unlabeled legacy gateway +namespaces (management labels, no instance label, `openshell-*` not +`openshell-db-*`) so missed-delete orphans from before the instance label are +not invisible to GC. Grace timer persisted on the `hypershell.redhat.io/gc-eligible-since` annotation (RFC3339) and cleared when a Gateway reappears. Sweep aborts entirely if Gateways cannot be listed, so a transient API failure never reaps a live namespace. A `GarbageCollected` Event is recorded in the control-plane namespace diff --git a/specs/platform/e2e-testing.spec.md b/specs/platform/e2e-testing.spec.md index 781e92af..6c6c795e 100644 --- a/specs/platform/e2e-testing.spec.md +++ b/specs/platform/e2e-testing.spec.md @@ -619,6 +619,7 @@ The `deploy/` directory SHALL use a kustomize base/overlay structure to support - WHEN `kustomize build deploy/openshift/` is executed - THEN the output SHALL include all base resources - AND OpenShift-specific resources: Routes for the API server, web console, and Keycloak with edge TLS termination; SecurityContextConstraints RoleBindings; `keycloak-allow-platform` NetworkPolicy so platform pods can reach Keycloak on TCP/8080 +- AND the API server Deployment SHALL set `API_ENV=development_oidc` (the overlay, not the lifecycle script) - AND `make openshift-up` SHALL rewrite overlay namespaces so `hypershell-system` maps to `OPENSHIFT_NAMESPACE` and `keycloak` maps to `${OPENSHIFT_NAMESPACE}-keycloak`, prefix cluster-scoped RBAC names with `${OPENSHIFT_NAMESPACE}-dev-`, and set the control plane `GATEWAY_API_BASE_DOMAIN` from the shared Gateway listener hostname #### Scenario: Base Resource Propagation diff --git a/specs/platform/openshell-gateway-namespace-gc.spec.md b/specs/platform/openshell-gateway-namespace-gc.spec.md index 8134f058..2df24ebf 100644 --- a/specs/platform/openshell-gateway-namespace-gc.spec.md +++ b/specs/platform/openshell-gateway-namespace-gc.spec.md @@ -54,8 +54,12 @@ Namespace creation and provisioning mechanics are defined in - `hypershell.redhat.io/instance=` Periodic garbage collection sweeps only namespaces owned by this instance whose names match the gateway prefix (`openshell-`) and excludes ManagedDatabase - namespaces (`openshell-db-`). A namespace owned by a different instance, or - lacking this instance's identity label, is never listed, annotated, or reaped. + namespaces (`openshell-db-`). A namespace owned by a different instance is + never listed, annotated, or reaped. A gateway namespace that carries both + management labels but no instance label is a pre-label leftover: each GC sweep + claims it for this instance (stamps `hypershell.redhat.io/instance`) and then + evaluates it as any other owned namespace. ManagedDatabase namespaces and + namespaces already labeled for another instance are never claimed. A Gateway pointed at a pre-existing or shared namespace can never cause that namespace to be reaped. - **Orphaned namespace** - a gateway namespace (matching the gateway prefix, not @@ -151,7 +155,18 @@ API server) for at least the grace period. The list selector SHALL include management labels, so a namespace created by another HyperShell instance on the same cluster is never observed as an orphan of this instance. If `HYPERSHELL_NAMESPACE` is empty, the reconciler SHALL abort the sweep rather than -list by the generic management labels alone. The sweep interval defaults to +list by the generic management labels alone. + +Before that instance-scoped list, each sweep SHALL claim unlabeled legacy +gateway namespaces: those that carry both management labels, have no +`hypershell.redhat.io/instance` key, and match the gateway prefix (not +`openshell-db-`). Claiming SHALL stamp this instance's identity and SHALL +NOT overwrite a label that already identifies a different instance. Unlabeled +ManagedDatabase namespaces SHALL NOT be claimed. After claiming, the sweep SHALL +evaluate those namespaces as owned by this instance (live Gateways are retained; +orphans start or continue the grace timer and are reaped after it elapses). If +claiming fails, the reconciler SHALL still sweep already-labeled namespaces. +The sweep interval defaults to 5 minutes and the grace period defaults to 10 minutes. Garbage collection SHALL be enabled by default and configurable without code changes via environment variables: @@ -182,12 +197,33 @@ database prefix, and carrying this instance's identity label). - THEN it SHALL NOT list, annotate, or delete instance A's namespaces - AND it SHALL NOT record them as orphaned of instance B -#### Scenario: Unlabeled legacy namespace is not swept +#### Scenario: Unlabeled legacy orphan is claimed then swept - GIVEN a namespace that carries the two management labels but not `hypershell.redhat.io/instance` +- AND its name is gateway-prefixed (`openshell-`, not `openshell-db-`) +- AND no live Gateway in this instance's API server maps to it - WHEN this instance's garbage-collection reconciler sweeps -- THEN it SHALL NOT list, annotate, or delete that namespace +- THEN it SHALL stamp `hypershell.redhat.io/instance` for this instance +- AND it SHALL treat the namespace as an orphan of this instance (stamp + `gc-eligible-since`, and delete after the grace period) + +#### Scenario: Live unlabeled gateway namespace is labeled but not orphaned + +- GIVEN a namespace that carries the two management labels but not + `hypershell.redhat.io/instance` +- AND a live Gateway in this instance's API server maps to it +- WHEN this instance's garbage-collection reconciler sweeps +- THEN it SHALL stamp `hypershell.redhat.io/instance` for this instance +- AND it SHALL NOT stamp `gc-eligible-since` or delete that namespace + +#### Scenario: Unlabeled ManagedDatabase namespace is not claimed + +- GIVEN a namespace named `openshell-db-` that carries the two management + labels but not `hypershell.redhat.io/instance` +- WHEN this instance's garbage-collection reconciler sweeps +- THEN it SHALL NOT stamp an instance label on that namespace +- AND it SHALL NOT annotate or delete that namespace #### Scenario: ManagedDatabase namespace is not reaped @@ -206,10 +242,11 @@ database prefix, and carrying this instance's identity label). - GIVEN the control plane was down when a Gateway is deleted, so the namespace was never reaped by the delete path -- AND the namespace carries this instance's identity label +- AND the namespace is either labeled for this instance or is an unlabeled + legacy namespace carrying both management labels - WHEN the control plane restarts and the garbage-collection reconciler sweeps -- THEN it SHALL observe the namespace as orphaned and, once the grace period has - elapsed, delete it +- THEN it SHALL observe the namespace as orphaned (claiming an unlabeled + leftover first) and, once the grace period has elapsed, delete it ### Requirement: Stamp This Instance's Identity on Namespaces It Manages @@ -220,9 +257,12 @@ two management labels. If the namespace already carries a different instance identity, the control plane SHALL NOT adopt, relabel, or delete it. Reconciling a live Gateway SHALL add the instance label to a legacy unlabeled namespace this instance is actively managing, so a later orphan can be reaped by this instance's -periodic GC. An empty `HYPERSHELL_NAMESPACE` SHALL NOT create or relabel a -managed namespace. Two controllers on the same cluster SHALL NOT share an -instance identity. +periodic GC. Each GC sweep SHALL also stamp this instance's identity onto +unlabeled legacy gateway namespaces (both management labels, no instance label, +gateway prefix) so a missed-delete orphan that never hits the live-reconcile +path becomes eligible for the orphan sweep. An empty `HYPERSHELL_NAMESPACE` +SHALL NOT create or relabel a managed namespace. Two controllers on the same +cluster SHALL NOT share an instance identity. #### Scenario: Created gateway namespace is labeled for this instance @@ -249,6 +289,15 @@ instance identity. - THEN it SHALL NOT overwrite the instance label - AND it SHALL NOT deploy into or delete that namespace +#### Scenario: Periodic GC backfills unlabeled legacy gateway namespaces + +- GIVEN a gateway-prefixed namespace that carries the two management labels + but no instance label +- WHEN the garbage-collection reconciler sweeps +- THEN it SHALL stamp `hypershell.redhat.io/instance` for this instance +- AND it SHALL NOT change an instance label that already identifies a different + instance + ### Requirement: Grace Period Prevents Premature Deletion The control plane SHALL NOT delete an orphaned namespace immediately. It SHALL @@ -384,9 +433,9 @@ real-time guarantee. | GC triggers on orphaning (no live Gateway), not on `phase` being `Degraded`/`Failed` | A gateway that still exists - even if unhealthy - is the health reconciler's and the operator's concern; only the absence of a backing Gateway unambiguously means the namespace is garbage. This avoids reaping a namespace an operator is still debugging. | | Exclude `openshell-db-*` managed namespaces from periodic GC | ManagedDatabase CNPG namespaces share the management labels but are owned by the ManagedDatabase reconciler; the stable `openshell-db-` prefix distinguishes them without requiring a label migration on existing gateway namespaces. | | Require this instance's identity label before periodic GC | Two HyperShell controllers on one cluster share the generic management labels. Without a value unique to that controller (`hypershell.redhat.io/instance=`), instance B's sweep treats instance A's live gateways as orphans (they are absent from B's API server) and would delete them after the grace period. The identity is the controller pod's namespace from the downward API so it cannot be a copied static string. | -| Leave unlabeled legacy namespaces out of periodic GC | Fail closed on coexistence: a namespace without this instance's identity might belong to another HyperShell that predates the label. Delete-driven cleanup of a Gateway from this API server may still reap an unlabeled managed namespace, because that path is keyed to this instance's own Gateway rather than a cluster-wide list. | +| Claim unlabeled legacy gateway namespaces at the start of each GC sweep | The instance label was added after namespaces already existed. `EnsureManagedNamespace` only stamps live Gateways; a missed-delete orphan never hits that path and would leak forever if unlabeled namespaces stayed excluded from the sweep. Each sweep lists namespaces that carry both management labels and no instance label, stamps this instance on gateway-prefixed names, then proceeds with the instance-scoped orphan sweep. Namespaces already labeled for another instance are never claimed. Two HyperShell instances that both still have unlabeled namespaces on the same cluster (neither has been upgraded to stamp the instance label) are unsupported: upgrade both, or label namespaces before enabling GC. | | Require BOTH management labels plus this instance's identity before deleting | Defense in depth: even if a label selector over-returns, a namespace not created by this control-plane instance (another HyperShell, a shared namespace, or a pre-existing namespace) is never deleted by periodic GC. | -| Stamp the instance label on create and on live reconcile | Periodic GC can only see namespaces this instance labeled. Stamping at create covers new gateways; stamping on live reconcile migrates this instance's pre-label namespaces so they remain eligible for a later orphan sweep. | +| Stamp the instance label on create, on live reconcile, and on GC backfill | Periodic GC can only reap namespaces this instance labeled. Stamping at create covers new gateways; stamping on live reconcile migrates this instance's still-live pre-label namespaces; GC backfill migrates unlabeled leftovers that have no live Gateway so the missed-delete path can proceed. | | Grace period persisted on the namespace annotation | The delay must survive control-plane restarts; storing `gc-eligible-since` on the namespace makes the timer durable without a separate store. | | Abort the whole sweep if Gateways cannot be listed | An empty or failed Gateway list would make every managed namespace look orphaned; aborting is the only safe response to avoid mass reaping of live namespaces. | | Delete is best-effort and not gated on sandbox count | Deletion is idempotent - process the delete, remove the namespace, and if it is already gone consider the delete done. The sandbox count is a warning surfaced to the operator, not a backend precondition. | diff --git a/specs/platform/openshift-development.spec.md b/specs/platform/openshift-development.spec.md index d0011f34..a07c9e69 100644 --- a/specs/platform/openshift-development.spec.md +++ b/specs/platform/openshift-development.spec.md @@ -400,6 +400,16 @@ produce incorrect discovery and issuer URLs and break token validation. This mir the Kind configuration, where `KC_HOSTNAME` is host-only and only the issuer values carry `/realms/hypershell`. +The API server JWT environment SHALL live in `deploy/openshift/kustomization.yaml` +with the other JWT and RBAC overlay patches, not in the lifecycle script. The +default `development` environment force-disables JWT after flag parsing, so +`--enable-jwt=true` is silent unless `API_ENV=development_oidc`. Route-derived +values (Keycloak `KC_HOSTNAME`, console redirect URIs, gateway OIDC issuer) remain +script-applied after Routes are assigned, because those hosts are not known at +kustomize-build time. The overlay SHALL re-declare any base container env the +JSON6902 env-array replace would otherwise drop, including +`HYPERSHELL_SERVICE_ACCOUNT_PROVISIONER_ADDR`. + This spec defines only where Keycloak lands. The broader isolation of other non-request-serving components (for example the database and observability) into their own namespaces is out of scope here and belongs to a separate spec. @@ -415,6 +425,14 @@ their own namespaces is out of scope here and belongs to a separate spec. - AND the current oc project is `alice` after Keycloak is applied - AND the OIDC issuer points at the Keycloak route in `alice-keycloak` +#### Scenario: API_ENV is declared in the OpenShift overlay + +- GIVEN `deploy/openshift/kustomization.yaml` is built +- WHEN the rendered API server Deployment is inspected +- THEN it SHALL set `API_ENV=development_oidc` +- AND it SHALL retain `HYPERSHELL_SERVICE_ACCOUNT_PROVISIONER_ADDR` +- AND `make openshift-up` SHALL NOT set `API_ENV` with `oc set env` + #### Scenario: Platform workloads can reach Keycloak across the namespace group `oc new-project` installs default-deny Ingress NetworkPolicies in each @@ -949,17 +967,24 @@ unprefixed ClusterRole `hypershell-controller` or ClusterRoleBinding cluster, such as stage. Built-in ClusterRoles whose names start with `system:` SHALL NOT be renamed. -When `OPENSHIFT_USE_EXISTING_CLUSTERROLE` is `true`, the command SHALL look up -ClusterRole `hypershell-controller` and SHALL apply a ClusterRoleBinding named +If the prefixed ClusterRole apply is Forbidden (Kubernetes escalation +prevention: the ClusterRole grants verbs the current user does not hold, for +example `routes/custom-host`) and ClusterRole `hypershell-controller` exists, +the command SHALL apply a ClusterRoleBinding named `${OPENSHIFT_NAMESPACE}-dev-hypershell-controller` whose `roleRef` is that existing ClusterRole and whose subject is this environment's controller -service account. It SHALL NOT create a ClusterRole, and it SHALL NOT look for -`hypershell-controller-scc-bind`. If ClusterRole `hypershell-controller` is -missing, the command SHALL fail. +service account. It SHALL NOT create a ClusterRole on that path, and it SHALL +NOT look for `hypershell-controller-scc-bind`. ClusterRoleBinding `roleRef` is +immutable, so if a ClusterRoleBinding of that name already points at a +different ClusterRole (for example a previous apply created the binding before +the ClusterRole was rejected), the command SHALL delete and recreate it so the +binding points at `hypershell-controller`. The command SHALL apply the +ClusterRole before the ClusterRoleBinding so a failed ClusterRole create does +not leave a binding that cannot be retargeted. Kubernetes escalation prevention forbids a typical developer from granting -those ClusterRoles, so when ClusterRole or ClusterRoleBinding apply is -Forbidden, the command SHALL warn and continue with the rest of the stack. +those ClusterRoles, so when both the prefixed ClusterRole and the existing-role +fallback fail, the command SHALL warn and continue with the rest of the stack. Gateways and sandboxes will not provision until this environment's controller is bound. `make openshift-down` SHALL delete this environment's `${OPENSHIFT_NAMESPACE}-dev-*` ClusterRoles and ClusterRoleBindings and SHALL @@ -981,14 +1006,14 @@ namespace. - AND the controller can create the per-namespace privileged RoleBinding for a sandbox because it was granted `bind` - AND the deployment does not attempt to grant the cluster-scoped `bind` through a namespace-scoped RoleBinding -#### Scenario: Shared cluster ClusterRole workaround +#### Scenario: Prefixed ClusterRole is forbidden, existing ClusterRole is used -- GIVEN ClusterRole `hypershell-controller` already exists on the cluster -- AND `OPENSHIFT_USE_EXISTING_CLUSTERROLE=true` +- GIVEN the developer cannot create ClusterRole `${OPENSHIFT_NAMESPACE}-dev-hypershell-controller` because of escalation prevention +- AND ClusterRole `hypershell-controller` already exists on the cluster - AND the developer can create ClusterRoleBindings prefixed with `${OPENSHIFT_NAMESPACE}-dev-` - WHEN the developer runs `make openshift-up` -- THEN the command looks up ClusterRole `hypershell-controller` -- AND applies ClusterRoleBinding `${OPENSHIFT_NAMESPACE}-dev-hypershell-controller` with `roleRef` `hypershell-controller` +- THEN the command applies ClusterRoleBinding `${OPENSHIFT_NAMESPACE}-dev-hypershell-controller` with `roleRef` `hypershell-controller` +- AND if that ClusterRoleBinding already pointed at a different ClusterRole, the command replaces it - AND the command does not create a ClusterRole - AND the command does not create or patch ClusterRoleBinding `hypershell-controller` From bde0e275351d01f187c1fe99bf59104b74b3f161 Mon Sep 17 00:00:00 2001 From: Kyle Squizzato Date: Wed, 2 Sep 2026 13:25:55 -0700 Subject: [PATCH 08/14] docs(e2e): Update DEVELOPMENT.md to include OpenShift scripts Signed-off-by: Kyle Squizzato --- DEVELOPMENT.md | 192 +++++++++++------- .../serviceAccounts/provisioner_client.go | 2 + deploy/base/api-server.yaml | 6 +- deploy/openshift/kustomization.yaml | 2 +- scripts/cluster/lib_test.sh | 5 +- specs/platform/openshift-development.spec.md | 11 +- 6 files changed, 134 insertions(+), 84 deletions(-) diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md index 30c00414..76c007e8 100644 --- a/DEVELOPMENT.md +++ b/DEVELOPMENT.md @@ -1,26 +1,48 @@ # Local Development Environment -HyperShell provides a single-command local development environment. Kind -([Kind](https://kind.sigs.k8s.io/), Kubernetes in Docker) is the default -single-tenant path. OpenShift uses the same make-target pattern -(`make openshift-up`, `make openshift--up`) against an existing -cluster and an ephemeral namespace group. The environment deploys all platform -components -- API server, control plane, and web console -- so developers can -test changes end-to-end. +HyperShell deploys the same stack (API server, control plane, web console, +Keycloak) on Kind or on an existing OpenShift cluster. Targets follow the same +shape: `make kind-` or `make openshift-`. Kind creates a local +cluster. OpenShift never creates a cluster; it deploys into the oc project you +select. + +## Contents + +- [Prerequisites](#prerequisites) +- [Kind Development](#kind-development) + - [Quickstart](#quickstart) + - [Kind make targets](#kind-make-targets) + - [Per-component swap](#per-component-swap) + - [Keycloak](#keycloak) + - [Kind environment variables](#kind-environment-variables) +- [OpenShift Development](#openshift-development) + - [OpenShift quickstart](#openshift-quickstart) + - [OpenShift make targets](#openshift-make-targets) + - [OpenShift per-component swap](#openshift-per-component-swap) + - [OpenShift environment variables](#openshift-environment-variables) +- [Gateway Access](#gateway-access) +- [Troubleshooting](#troubleshooting) ## Prerequisites | Tool | Purpose | Install | |------|---------|---------| | [Docker](https://docs.docker.com/get-docker/) or [Podman](https://podman.io/docs/installation) | Container engine | OS package manager | -| [Kind](https://kind.sigs.k8s.io/docs/user/quick-start/#installation) | Local Kubernetes clusters | `brew install kind` | -| [kubectl](https://kubernetes.io/docs/tasks/tools/) | Kubernetes CLI | `brew install kubectl` | +| [Kind](https://kind.sigs.k8s.io/docs/user/quick-start/#installation) | Local Kubernetes clusters (Kind path) | `brew install kind` | +| [kubectl](https://kubernetes.io/docs/tasks/tools/) | Kubernetes CLI (Kind path) | `brew install kubectl` | +| [oc](https://docs.openshift.com/container-platform/latest/cli_reference/openshift_cli/getting-started-cli.html) | OpenShift CLI (OpenShift path) | [install oc](https://docs.openshift.com/container-platform/latest/cli_reference/openshift_cli/getting-started-cli.html) | | [cloud-provider-kind](https://github.com/kubernetes-sigs/cloud-provider-kind) | LoadBalancer + Gateway API for Kind | `brew install cloud-provider-kind` | The container engine is auto-detected (Podman preferred). Override with `CONTAINER_ENGINE=docker` or `CONTAINER_ENGINE=podman`. -## Quickstart +## Kind Development + +Kind ([Kubernetes in Docker](https://kind.sigs.k8s.io/)) is the default +single-tenant path. `make kind-up` creates the cluster and deploys every +platform component so you can test changes end to end. + +### Quickstart ```bash make kind-up @@ -57,7 +79,30 @@ host port 443 to cloud-provider-kind's ephemeral Gateway port. The TLS certificate is self-signed -- trust it in your browser or use `curl --cacert`. -## Per-Component Swap +### Kind make targets + +`make kind-up` is idempotent: running it again on an existing cluster +reapplies manifests and waits for readiness. Swapped components are preserved. + +| Target | Use | +|--------|-----| +| `make kind-up` | Create the Kind cluster, install prerequisites, deploy the stack, seed sample resources, and wait until ready. | +| `make kind-down` | Remove the `hypershell-system` namespace and its resources. Leaves the Kind cluster running. | +| `make kind-teardown` | Destroy the Kind cluster and stop cloud-provider-kind. | +| `make kind-status` | Show cluster info, pods, services, and which components are swapped. | +| `make kind-seed` | Re-run ManagedCluster, GatewayRelease, ManagedDatabase, and Gateway seeding. `kind-up` already seeds unless `SKIP_SEED=true`. | +| `make kind-prereqs` | Build the pinned `cloud-provider-kind` binary into `bin/`. `kind-up` runs this; use it alone when the binary is missing. | +| `make kind-env` | Print `export` statements for the current Kind make variables. | +| `make kind-fix-ports` | Re-establish host port 443 forwarding to the Gateway's ephemeral port. | +| `make kind-gateway-trust` | Write the cluster CA to `bin/hypershell-ca.crt` and print `export SSL_CERT_FILE=...` for the openshell CLI. Run `eval "$(make kind-gateway-trust)"`. | +| `make kind-api-server-up` | Build the API server from the working tree and swap it into the cluster. | +| `make kind-api-server-down` | Revert the API server to the baseline registry image. | +| `make kind-control-plane-up` | Build the control plane from the working tree and swap it into the cluster. | +| `make kind-control-plane-down` | Revert the control plane to the baseline registry image. | +| `make kind-web-console-up` | Hot-reload the web console (default), or build and swap a full image when `KIND_HOT_RELOAD=false`. | +| `make kind-web-console-down` | Revert the web console to the baseline registry image. | + +### Per-component swap Baseline images are pulled from the container registry. To build all baseline images locally instead (e.g. when registry access is unavailable), run: @@ -108,7 +153,7 @@ make kind-status Shows which components are running local builds vs. baseline images. Swap state is tracked in `.kind-swaps` (gitignored). -## Hot Reload +### Hot reload The web console supports hot reload by default. When you run `make kind-web-console-up`, the host source directory is mounted into the @@ -125,7 +170,7 @@ Hot reload is only supported for the web console. The API server and control plane are Go services that require a full rebuild (`make kind-api-server-up` / `make kind-control-plane-up`). -## Keycloak +### Keycloak The local Keycloak instance mirrors the downstream Keycloak topology used in production. @@ -192,7 +237,7 @@ KIND_KEYCLOAK_URL=https://keycloak.example.com/realms/hypershell make kind-up This skips the local Keycloak deployment and points the gateway OIDC issuer at the external URL. -## OIDC Authentication +### OIDC authentication The Kind cluster runs with OIDC authentication enabled. Keycloak is deployed as the identity provider and all components are configured for JWT validation and @@ -250,7 +295,7 @@ curl -s -H "Authorization: Bearer ${TOKEN}" \ https://api.hypershell.localhost/api/hypershell/v1/gateways ``` -## Private Registry Pull Secret +### Private registry pull secret If your baseline images live in a private registry, provide a pull secret: @@ -261,7 +306,7 @@ KIND_PULL_SECRET=/path/to/pull-secret.yaml make kind-up The YAML file is applied into the target namespace with `kubectl apply`. It should contain a `kubernetes.io/dockerconfigjson` Secret. -## Offline Development +### Offline development Build all component images from the local working tree instead of pulling from the container registry: @@ -274,26 +319,41 @@ This builds api-server, control-plane, and web-console images locally and loads them into Kind. The Dockerfiles drop the local `rh-trex-ai` replace directive at build time, so no external dependency checkout is needed. -## Cluster Lifecycle - -```bash -make kind-up # Create cluster + deploy everything -make kind-down # Remove namespace and its resources -make kind-teardown # Destroy Kind cluster entirely -make kind-status # Show cluster info, pods, services, swap state -``` +### Kind environment variables -`make kind-up` is idempotent -- running it again on an existing cluster -reapplies manifests and waits for readiness. Swapped components are preserved. +| Variable | Default | Description | +|----------|---------|-------------| +| `KIND_CLUSTER_NAME` | `hypershell-dev` | Kind cluster name | +| `KIND_NAMESPACE` | `hypershell-system` | Target namespace for swap and teardown | +| `KIND_HOT_RELOAD` | `true` | Hot reload for the web console | +| `KIND_HOST_MOUNT_PATH` | Repository root | Host directory mounted into Kind nodes | +| `KIND_KEYCLOAK_URL` | (unset) | External Keycloak URL; skips local deploy | +| `KEYCLOAK_OIDC_ISSUER` | `https://keycloak.hypershell.localhost/realms/hypershell` | OIDC issuer URL | +| `KIND_PULL_SECRET` | (unset) | Path to pull secret YAML for private registries | +| `IMAGE_REGISTRY` | `quay.io/redhat-services-prod/hcm-eng-prod-tenant/hypershell-main` | Container registry for baseline images | +| `IMAGE_TAG` | `latest` | Image tag for baseline images | +| `LOCAL_IMAGES` | (unset) | Set to `true` to build baseline images from the working tree | +| `CONTAINER_ENGINE` | Auto-detected | `podman` or `docker` | +| `GATEWAY_API_VERSION` | `v1.5.1` | Gateway API CRD version | +| `CLOUD_PROVIDER_KIND_REPO` | `https://github.com/squizzi/cloud-provider-kind.git` | cloud-provider-kind git repo | +| `CLOUD_PROVIDER_KIND_BRANCH` | `hypershell` | cloud-provider-kind branch to build | +| `CERT_MANAGER_VERSION` | `v1.21.1` | cert-manager version | +| `KIND_DB_IMAGE` | `registry.access.redhat.com/hi/postgresql:18.4@sha256:9b19...` | Database image for Gateway; override for OSS dev | +| `KIND_NO_SUDO` | (unset) | Set to `true` to skip sudo operations | +| `KIND_DNS_PORT` | `5553` | Host port for CoreDNS container | +| `SKIP_SEED` | (unset) | Set to `true` to skip seeding on `make kind-up`. `KIND_SKIP_SEED` is still accepted. | +| `SEED_STRICT` | (unset) | Set to `true` to fail `make kind-up` / `make kind-seed` if seeding is incomplete. `KIND_SEED_STRICT` is still accepted. | ## OpenShift Development -`make openshift-up` deploys the same stack into an isolated namespace group on -an OpenShift cluster selected by the current kubeconfig context. It does not -create the cluster. An administrator must already have provisioned the shared -Gateway, GatewayClass, certificate issuer, and wildcard certificate (see +`make openshift-up` deploys the same stack into the current oc project on an +OpenShift cluster selected by kubeconfig. It does not create the cluster. An +administrator must already have provisioned the shared Gateway, GatewayClass, +certificate issuer, and wildcard certificate (see `deploy/openshift/infrastructure/GATEWAY-SETUP.md`). +### OpenShift quickstart + The platform namespace is the current oc project. Select it first, then bring the environment up: @@ -314,6 +374,23 @@ companion Keycloak namespace `${name}-keycloak` stays within the 63-character limit. If no project is selected and `OPENSHIFT_NAMESPACE` is unset, the command stops with an error. +### OpenShift make targets + +| Target | Use | +|--------|-----| +| `make openshift-up` | Deploy the stack into the current oc project (`OPENSHIFT_NAMESPACE` override) and companion `${name}-keycloak`. Does not create an OpenShift cluster. Seeds unless `SKIP_SEED=true`. | +| `make openshift-down` | Delete the platform and Keycloak projects. If project deletion is forbidden, strip HyperShell resources and leave the projects. | +| `make openshift-teardown` | Same as `openshift-down`. There is no OpenShift cluster to destroy. | +| `make openshift-status` | Show namespaces, pods, Routes, the shared Gateway, and swap state. | +| `make openshift-seed` | Re-run ManagedCluster, GatewayRelease, ManagedDatabase, and Gateway seeding via API and Keycloak Routes from this machine. `openshift-up` already seeds unless `SKIP_SEED=true`. | +| `make openshift-api-server-up` | Build, push an immutable image to the internal registry, and swap the API server. | +| `make openshift-api-server-down` | Revert the API server to the baseline registry image. | +| `make openshift-control-plane-up` | Build, push, and swap the control plane. | +| `make openshift-control-plane-down` | Revert the control plane to the baseline registry image. | +| `make openshift-web-console-up` | Build, push, and swap the web console. There is no Kind-style Vite hot reload. | +| `make openshift-web-console-down` | Revert the web console to the baseline registry image. | +| `make openshift-test` | Run `scripts/cluster/lib_test.sh` on the laptop (overlay rewrite tests). Does not talk to a cluster. | + `make openshift-up` deploys into the project you selected. It does not ask for confirmation, and it does not require permission to label the namespace. When the account can patch namespaces, the scripts stamp HyperShell ownership @@ -360,7 +437,7 @@ and the companion `${name}-keycloak` project. If project deletion is forbidden, they delete HyperShell resources inside both projects (including unlabeled Keycloak) and leave the projects. Labels are not required. -### Per-component swap +### OpenShift per-component swap ```bash make openshift-api-server-up @@ -376,60 +453,19 @@ and the exact image each one uses. Swap state is tracked per namespace in `.openshift-swaps/` (gitignored). A subsequent `make openshift-up` preserves active swaps. -## Environment Variable Reference +### OpenShift environment variables | Variable | Default | Description | |----------|---------|-------------| -| `KIND_CLUSTER_NAME` | `hypershell-dev` | Kind cluster name | -| `KIND_NAMESPACE` | `hypershell-system` | Target namespace for swap/teardown | -| `KIND_HOT_RELOAD` | `true` | Hot reload for web console | -| `KIND_HOST_MOUNT_PATH` | Repository root | Host directory mounted into Kind nodes | -| `KIND_KEYCLOAK_URL` | (unset) | External Keycloak URL; skips local deploy | -| `KEYCLOAK_OIDC_ISSUER` | `https://keycloak.hypershell.localhost/realms/hypershell` | OIDC issuer URL | -| `KIND_PULL_SECRET` | (unset) | Path to pull secret YAML for private registries | -| `IMAGE_REGISTRY` | `quay.io/redhat-services-prod/hcm-eng-prod-tenant/hypershell-main` | Container registry | -| `IMAGE_TAG` | `latest` | Image tag for baseline images | -| `LOCAL_IMAGES` | (unset) | Set to `true` for offline baseline builds | -| `CONTAINER_ENGINE` | Auto-detected | `podman` or `docker` | -| `GATEWAY_API_VERSION` | `v1.5.1` | Gateway API CRD version | -| `CLOUD_PROVIDER_KIND_REPO` | `https://github.com/squizzi/cloud-provider-kind.git` | cloud-provider-kind git repo | -| `CLOUD_PROVIDER_KIND_BRANCH` | `hypershell` | cloud-provider-kind branch to build | -| `CERT_MANAGER_VERSION` | `v1.21.1` | cert-manager version | -| `KIND_DB_IMAGE` | `registry.access.redhat.com/hi/postgresql:18.4@sha256:9b19...` | Database image for Gateway; override for OSS dev | -| `KIND_NO_SUDO` | (unset) | Set to `true` to skip sudo operations | -| `KIND_DNS_PORT` | `5553` | Host port for CoreDNS container | | `OPENSHIFT_NAMESPACE` | `oc project -q` | Override for the platform namespace. Unset, the current oc project is used. Max 54 chars; Keycloak lands in `${name}-keycloak`. | -| `SKIP_SEED` | (unset) | Set to `true` to skip ManagedCluster/GatewayRelease/ManagedDatabase/Gateway seeding on `make kind-up` and `make openshift-up`. `KIND_SKIP_SEED` is still accepted. | -| `SEED_STRICT` | (unset) | Set to `true` to fail `make kind-up` / `make kind-seed` / `make openshift-up` if seeding is incomplete. `KIND_SEED_STRICT` is still accepted. | | `GATEWAY_API_GATEWAY_NAME` | `openshell-grpc-gateway` | Pre-existing shared Gateway name | | `GATEWAY_API_GATEWAY_NAMESPACE` | `openshift-ingress` | Namespace of the shared Gateway | | `OPENSHIFT_IMAGE_REGISTRY` | `oc registry info` | Registry used to push swapped images | - -## Make Targets - -| Target | Description | -|--------|-------------| -| `make kind-up` | Create cluster + prerequisites + deploy + wait | -| `make kind-down` | Remove namespace and its resources | -| `make kind-teardown` | Destroy Kind cluster, stop cloud-provider-kind | -| `make kind-status` | Show cluster info, pods, services, swap state | -| `make kind-api-server-up` | Build + swap API server from working tree | -| `make kind-api-server-down` | Revert API server to baseline image | -| `make kind-control-plane-up` | Build + swap control plane from working tree | -| `make kind-control-plane-down` | Revert control plane to baseline image | -| `make kind-web-console-up` | Hot reload (default) or build + swap web console | -| `make kind-web-console-down` | Revert web console to baseline image | -| `make kind-fix-ports` | Re-establish host port forwarding (443) | -| `make openshift-up` | Deploy into an ephemeral OpenShift namespace group | -| `make openshift-down` | Remove the namespace group (platform project and `${name}-keycloak`) | -| `make openshift-teardown` | Same as `openshift-down` (OpenShift has no cluster to destroy) | -| `make openshift-status` | Show namespaces, pods, Routes, Gateway, swap state | -| `make openshift-api-server-up` | Build, push, and swap API server from working tree | -| `make openshift-api-server-down` | Revert API server to baseline image | -| `make openshift-control-plane-up` | Build, push, and swap control plane from working tree | -| `make openshift-control-plane-down` | Revert control plane to baseline image | -| `make openshift-web-console-up` | Build, push, and swap web console from working tree | -| `make openshift-web-console-down` | Revert web console to baseline image | +| `IMAGE_REGISTRY` | `quay.io/redhat-services-prod/hcm-eng-prod-tenant/hypershell-main` | Container registry for baseline images | +| `IMAGE_TAG` | `latest` | Image tag for baseline images | +| `CONTAINER_ENGINE` | Auto-detected | `podman` or `docker` | +| `SKIP_SEED` | (unset) | Set to `true` to skip seeding on `make openshift-up`. `KIND_SKIP_SEED` is still accepted. | +| `SEED_STRICT` | (unset) | Set to `true` to fail `make openshift-up` / `make openshift-seed` if seeding is incomplete. `KIND_SEED_STRICT` is still accepted. | ## Gateway Access diff --git a/components/api-server/plugins/serviceAccounts/provisioner_client.go b/components/api-server/plugins/serviceAccounts/provisioner_client.go index 5f83942e..e5109646 100644 --- a/components/api-server/plugins/serviceAccounts/provisioner_client.go +++ b/components/api-server/plugins/serviceAccounts/provisioner_client.go @@ -30,6 +30,8 @@ func newControlPlaneProvisionerFromEnvironment() (ServiceAccountProvisioner, err if address == "" { return nil, nil } + // grpc-go DNS does not apply kube-DNS search domains. Callers must pass a + // cluster-local FQDN (see deploy/base/api-server.yaml). connection, err := grpc.NewClient(address, grpc.WithTransportCredentials(insecure.NewCredentials())) if err != nil { return nil, fmt.Errorf("create control-plane provisioner client: %w", err) diff --git a/deploy/base/api-server.yaml b/deploy/base/api-server.yaml index 56825dae..d503669a 100644 --- a/deploy/base/api-server.yaml +++ b/deploy/base/api-server.yaml @@ -89,8 +89,12 @@ spec: # Plaintext in-cluster gRPC to the control-plane provisioner. A # NetworkPolicy restricts port 9443 to this pod; the channel mirrors # the sibling control-plane -> api-server watch channel (no mTLS). + # Use a cluster-local FQDN. grpc-go's DNS resolver does not apply + # kube-DNS search domains, so the short name hypershell-controller + # fails even when libc lookup (nc) succeeds. OpenShift rewrite maps + # hypershell-system to OPENSHIFT_NAMESPACE. - name: HYPERSHELL_SERVICE_ACCOUNT_PROVISIONER_ADDR - value: "hypershell-controller:9443" + value: "hypershell-controller.hypershell-system.svc.cluster.local:9443" - name: DB_SSLMODE value: require startupProbe: diff --git a/deploy/openshift/kustomization.yaml b/deploy/openshift/kustomization.yaml index 0ebebfe6..19352866 100644 --- a/deploy/openshift/kustomization.yaml +++ b/deploy/openshift/kustomization.yaml @@ -91,7 +91,7 @@ patches: # Re-declare the base env: this JSON6902 add REPLACES the whole env # array, so the base values would otherwise be dropped. - name: HYPERSHELL_SERVICE_ACCOUNT_PROVISIONER_ADDR - value: "hypershell-controller:9443" + value: "hypershell-controller.hypershell-system.svc.cluster.local:9443" - name: DB_SSLMODE value: require - name: JWK_CERT_URL diff --git a/scripts/cluster/lib_test.sh b/scripts/cluster/lib_test.sh index fb9a2418..3755c069 100755 --- a/scripts/cluster/lib_test.sh +++ b/scripts/cluster/lib_test.sh @@ -673,11 +673,12 @@ if bad: FAIL=$((FAIL + 1)) echo 'FAIL: OpenShift overlay does not set API_ENV=development_oidc' fi - if grep -q 'name: HYPERSHELL_SERVICE_ACCOUNT_PROVISIONER_ADDR' "${os_out}"; then + if grep -q 'name: HYPERSHELL_SERVICE_ACCOUNT_PROVISIONER_ADDR' "${os_out}" \ + && grep -q 'hypershell-controller.alice.svc.cluster.local:9443' "${os_out}"; then PASS=$((PASS + 1)) else FAIL=$((FAIL + 1)) - echo 'FAIL: OpenShift overlay dropped HYPERSHELL_SERVICE_ACCOUNT_PROVISIONER_ADDR' + echo 'FAIL: OpenShift overlay dropped HYPERSHELL_SERVICE_ACCOUNT_PROVISIONER_ADDR FQDN' fi else FAIL=$((FAIL + 1)) diff --git a/specs/platform/openshift-development.spec.md b/specs/platform/openshift-development.spec.md index a07c9e69..610e097d 100644 --- a/specs/platform/openshift-development.spec.md +++ b/specs/platform/openshift-development.spec.md @@ -408,7 +408,13 @@ values (Keycloak `KC_HOSTNAME`, console redirect URIs, gateway OIDC issuer) rema script-applied after Routes are assigned, because those hosts are not known at kustomize-build time. The overlay SHALL re-declare any base container env the JSON6902 env-array replace would otherwise drop, including -`HYPERSHELL_SERVICE_ACCOUNT_PROVISIONER_ADDR`. +`HYPERSHELL_SERVICE_ACCOUNT_PROVISIONER_ADDR`. That address SHALL be the +cluster-local FQDN +`hypershell-controller.${OPENSHIFT_NAMESPACE}.svc.cluster.local:9443` +(the overlay stores `hypershell-system`; namespace rewrite substitutes the +assigned project). grpc-go does not apply kube-DNS search domains, so the +short name `hypershell-controller:9443` fails in-cluster even when `nc` to +that short name succeeds. This spec defines only where Keycloak lands. The broader isolation of other non-request-serving components (for example the database and observability) into @@ -430,7 +436,8 @@ their own namespaces is out of scope here and belongs to a separate spec. - GIVEN `deploy/openshift/kustomization.yaml` is built - WHEN the rendered API server Deployment is inspected - THEN it SHALL set `API_ENV=development_oidc` -- AND it SHALL retain `HYPERSHELL_SERVICE_ACCOUNT_PROVISIONER_ADDR` +- AND it SHALL retain `HYPERSHELL_SERVICE_ACCOUNT_PROVISIONER_ADDR` as the + cluster-local FQDN for this project's controller Service - AND `make openshift-up` SHALL NOT set `API_ENV` with `oc set env` #### Scenario: Platform workloads can reach Keycloak across the namespace group From e10ef4408cc1484b418e7778743dcd429a595577 Mon Sep 17 00:00:00 2001 From: Kyle Squizzato Date: Wed, 2 Sep 2026 14:15:22 -0700 Subject: [PATCH 09/14] fix(e2e): auto-detect e2e driver and harden OpenShift deployment readiness --- DEVELOPMENT.md | 2 +- Makefile | 5 +- .../internal/gateway/namespace.go | 11 +-- .../internal/gateway/namespace_test.go | 28 +++++++- .../internal/reconciler/namespace.go | 8 ++- deploy/base/controller.yaml | 13 ++++ deploy/openshift/kustomization.yaml | 6 ++ scripts/cluster/drivers/openshift.sh | 70 ++++++++++--------- scripts/cluster/lib_test.sh | 39 +++++++++++ specs/platform/e2e-testing.spec.md | 62 ++++++++-------- .../openshell-gateway-namespace-gc.spec.md | 51 ++++++-------- specs/platform/openshift-development.spec.md | 57 ++++++++++++--- tests/e2e/e2e-openshell.sh | 28 ++++++-- 13 files changed, 262 insertions(+), 118 deletions(-) diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md index 76c007e8..92beda63 100644 --- a/DEVELOPMENT.md +++ b/DEVELOPMENT.md @@ -378,7 +378,7 @@ command stops with an error. | Target | Use | |--------|-----| -| `make openshift-up` | Deploy the stack into the current oc project (`OPENSHIFT_NAMESPACE` override) and companion `${name}-keycloak`. Does not create an OpenShift cluster. Seeds unless `SKIP_SEED=true`. | +| `make openshift-up` | Deploy the stack into the current oc project (`OPENSHIFT_NAMESPACE` override) and companion `${name}-keycloak`. Does not create an OpenShift cluster. Waits for component rollouts, then seeds unless `SKIP_SEED=true`. | | `make openshift-down` | Delete the platform and Keycloak projects. If project deletion is forbidden, strip HyperShell resources and leave the projects. | | `make openshift-teardown` | Same as `openshift-down`. There is no OpenShift cluster to destroy. | | `make openshift-status` | Show namespaces, pods, Routes, the shared Gateway, and swap state. | diff --git a/Makefile b/Makefile index 5c8b60cb..a917c9c1 100644 --- a/Makefile +++ b/Makefile @@ -525,10 +525,9 @@ E2E_INFRA_DRIVER ?= kind .PHONY: e2e e2e: @echo "" - @echo "==> Running E2E tests ($(E2E_INFRA_DRIVER))" + @echo "==> Running E2E tests" @echo "" - @E2E_INFRA_DRIVER=$(E2E_INFRA_DRIVER) \ - E2E_PROVISION_TIMEOUT=300 \ + @E2E_PROVISION_TIMEOUT=300 \ E2E_SANDBOX_TIMEOUT=180 \ bash tests/e2e/e2e-openshell.sh diff --git a/components/control-plane/internal/gateway/namespace.go b/components/control-plane/internal/gateway/namespace.go index e81e1125..f97cebd5 100644 --- a/components/control-plane/internal/gateway/namespace.go +++ b/components/control-plane/internal/gateway/namespace.go @@ -53,14 +53,17 @@ const ( ) // ManagedNamespaceSelector selects namespaces created by this control-plane -// instance. An empty instance matches nothing: listing by the generic management +// instance. An empty instance is an error: listing by the generic management // labels alone would treat every other HyperShell on the cluster as an orphan. -func ManagedNamespaceSelector(instance string) string { +// The returned selector is still a valid Kubernetes label selector that matches +// nothing, so a caller that ignores the error cannot trigger an HTTP 400 or +// list every namespace (an empty LabelSelector would). +func ManagedNamespaceSelector(instance string) (string, error) { if instance == "" { - return InstanceLabel + "=__no-such-instance__" + return InstanceLabel + "=no-such-instance", fmt.Errorf("instance identity is empty; refusing to build a managed-namespace selector") } return fmt.Sprintf("%s=%s,%s=%s,%s=%s", - ManagedLabel, ManagedLabelValue, ManagedByLabel, ManagedByValue, InstanceLabel, instance) + ManagedLabel, ManagedLabelValue, ManagedByLabel, ManagedByValue, InstanceLabel, instance), nil } // LegacyUnlabeledSelector selects HyperShell-managed namespaces that predate the diff --git a/components/control-plane/internal/gateway/namespace_test.go b/components/control-plane/internal/gateway/namespace_test.go index e9d6618e..3a4259a7 100644 --- a/components/control-plane/internal/gateway/namespace_test.go +++ b/components/control-plane/internal/gateway/namespace_test.go @@ -2,12 +2,15 @@ package gateway import ( "context" + "strings" "testing" "time" corev1 "k8s.io/api/core/v1" k8serrors "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/labels" + "k8s.io/apimachinery/pkg/util/validation" "k8s.io/client-go/kubernetes/fake" ) @@ -33,12 +36,33 @@ func managedNamespaceForInstance(name, instance string, annotations map[string]s } func TestManagedNamespaceSelector(t *testing.T) { - got := ManagedNamespaceSelector("alice") + got, err := ManagedNamespaceSelector("alice") + if err != nil { + t.Fatalf("ManagedNamespaceSelector() error = %v", err) + } want := "hypershell.redhat.io/managed=true,app.kubernetes.io/managed-by=hypershell-control-plane,hypershell.redhat.io/instance=alice" if got != want { t.Errorf("ManagedNamespaceSelector() = %q, want %q", got, want) } - if ManagedNamespaceSelector("") == ManagedNamespaceSelector("hypershell") { + empty, err := ManagedNamespaceSelector("") + if err == nil { + t.Fatal("ManagedNamespaceSelector(\"\") error = nil, want error") + } + if _, parseErr := labels.Parse(empty); parseErr != nil { + t.Fatalf("empty-instance selector %q is not a valid label selector: %v", empty, parseErr) + } + _, value, ok := strings.Cut(empty, "=") + if !ok { + t.Fatalf("empty-instance selector %q has no value", empty) + } + if msgs := validation.IsValidLabelValue(value); len(msgs) != 0 { + t.Fatalf("sentinel label value %q is invalid: %v", value, msgs) + } + real, err := ManagedNamespaceSelector("hypershell") + if err != nil { + t.Fatalf("ManagedNamespaceSelector(hypershell) error = %v", err) + } + if empty == real { t.Errorf("empty instance selector must not match a real instance") } } diff --git a/components/control-plane/internal/reconciler/namespace.go b/components/control-plane/internal/reconciler/namespace.go index 28e28597..66f18962 100644 --- a/components/control-plane/internal/reconciler/namespace.go +++ b/components/control-plane/internal/reconciler/namespace.go @@ -149,9 +149,15 @@ func (r *NamespaceGCReconciler) reconcileOnce(ctx context.Context) { } backfillCancel() + selector, err := gateway.ManagedNamespaceSelector(r.cpNamespace) + if err != nil { + tickErr = err + log.Printf("WARN namespace gc: %v", tickErr) + return + } listCtx, cancel := context.WithTimeout(ctx, namespaceListTimeout) namespaces, err := r.client.CoreV1().Namespaces().List(listCtx, metav1.ListOptions{ - LabelSelector: gateway.ManagedNamespaceSelector(r.cpNamespace), + LabelSelector: selector, }) cancel() if err != nil { diff --git a/deploy/base/controller.yaml b/deploy/base/controller.yaml index acfa419d..ca80ebc6 100644 --- a/deploy/base/controller.yaml +++ b/deploy/base/controller.yaml @@ -55,6 +55,19 @@ spec: ports: - containerPort: 9443 name: provisioner + startupProbe: + tcpSocket: + port: provisioner + periodSeconds: 2 + failureThreshold: 30 + livenessProbe: + tcpSocket: + port: provisioner + periodSeconds: 10 + readinessProbe: + tcpSocket: + port: provisioner + periodSeconds: 5 resources: requests: cpu: 100m diff --git a/deploy/openshift/kustomization.yaml b/deploy/openshift/kustomization.yaml index 19352866..57193169 100644 --- a/deploy/openshift/kustomization.yaml +++ b/deploy/openshift/kustomization.yaml @@ -33,6 +33,12 @@ patches: value: openshell.stage.example.com - name: GATEWAY_API_TLS_ISSUER_NAME value: hypershell-ca-issuer + # Shared Gateway listener is named grpc (see + # deploy/openshift/infrastructure/gateway.yaml). Console + # HTTPRoutes default sectionName to https, which yields + # NoMatchingParent on this cluster. + - name: GATEWAY_API_HTTP_LISTENER_NAME + value: grpc - name: OIDC_ISSUER valueFrom: secretKeyRef: diff --git a/scripts/cluster/drivers/openshift.sh b/scripts/cluster/drivers/openshift.sh index efbf3369..4afe586a 100755 --- a/scripts/cluster/drivers/openshift.sh +++ b/scripts/cluster/drivers/openshift.sh @@ -681,10 +681,21 @@ wait_for_route_host() { return 1 } +wait_for_named_rollout() { + local deployment="$1" + local ns="$2" + local timeout="${3:-180s}" + info "Waiting for ${deployment}..." + if ! oc_cli rollout status "deployment/${deployment}" -n "${ns}" --timeout="${timeout}"; then + error "${deployment} did not become ready in ${ns}" + oc_cli describe "deploy/${deployment}" -n "${ns}" | tail -30 || true + exit 1 + fi + success "${deployment} ready" +} + wait_for_keycloak() { - info "Waiting for Keycloak..." - oc_cli rollout status deployment/keycloak -n "${OPENSHIFT_KEYCLOAK_NAMESPACE}" --timeout=180s - success "Keycloak ready" + wait_for_named_rollout keycloak "${OPENSHIFT_KEYCLOAK_NAMESPACE}" } configure_oidc_from_routes() { @@ -732,6 +743,11 @@ configure_oidc_from_routes() { wait_for_deployments() { header "Readiness" + # Use `rollout status`, not `wait --for=condition=available`. With replicas=1 + # and the default rolling update the Deployment stays Available throughout a + # rollout (the old pod keeps serving until the new one is Ready), so + # `wait --for=condition=available` returns immediately after Route-derived + # `oc set env` and swap restore -- while a rollout is still in flight. wait_for_keycloak if [[ -n "$(oc_cli get cluster.postgresql.cnpg.io hypershell-db -n "${OPENSHIFT_NAMESPACE}" --ignore-not-found -o name 2>/dev/null || true)" ]]; then @@ -739,32 +755,12 @@ wait_for_deployments() { oc_cli wait --for=condition=Ready cluster/hypershell-db -n "${OPENSHIFT_NAMESPACE}" --timeout=300s \ || warn "CNPG cluster not Ready yet; API server will retry connections" elif oc_cli get deployment/hypershell-postgres -n "${OPENSHIFT_NAMESPACE}" >/dev/null 2>&1; then - info "Waiting for PostgreSQL deployment..." - if ! oc_cli rollout status deployment/hypershell-postgres -n "${OPENSHIFT_NAMESPACE}" --timeout=120s; then - warn "PostgreSQL rollout timed out. ReplicaSet events:" - oc_cli describe deploy/hypershell-postgres -n "${OPENSHIFT_NAMESPACE}" | tail -20 || true - oc_cli get events -n "${OPENSHIFT_NAMESPACE}" --field-selector involvedObject.kind=ReplicaSet \ - --sort-by='.lastTimestamp' 2>/dev/null | tail -10 || true - exit 1 - fi - success "PostgreSQL ready" + wait_for_named_rollout hypershell-postgres "${OPENSHIFT_NAMESPACE}" 120s fi - if ! is_openshift_swapped api-server; then - info "Waiting for API server..." - oc_cli rollout status deployment/hypershell-api-server -n "${OPENSHIFT_NAMESPACE}" --timeout=180s - success "API server ready" - fi - if ! is_openshift_swapped control-plane; then - info "Waiting for control plane..." - oc_cli rollout status deployment/hypershell-controller -n "${OPENSHIFT_NAMESPACE}" --timeout=180s - success "Control plane ready" - fi - if ! is_openshift_swapped web-console; then - info "Waiting for web console..." - oc_cli rollout status deployment/hypershell-web-console -n "${OPENSHIFT_NAMESPACE}" --timeout=180s - success "Web console ready" - fi + wait_for_named_rollout hypershell-api-server "${OPENSHIFT_NAMESPACE}" + wait_for_named_rollout hypershell-controller "${OPENSHIFT_NAMESPACE}" + wait_for_named_rollout hypershell-web-console "${OPENSHIFT_NAMESPACE}" } # Talk to OpenShift Routes from the developer machine. The API server image has @@ -936,16 +932,26 @@ seed_via_api() { fi if [[ -z "${seed_failed}" ]]; then + local db_provider="cnpg" + if ! cnpg_available; then + db_provider="deployment" + fi raw="$(api_exec GET /api/hypershell/v1/managed_databases)" http="$(printf '%s' "${raw}" | tail -1)" body="$(printf '%s' "${raw}" | sed '$d')" if [[ "${http}" == "200" ]]; then DATABASE_ID="$(extract_named_id "${body}" openshell-db)" fi - if [[ -z "${DATABASE_ID}" ]]; then - info "Creating ManagedDatabase..." + if [[ -n "${DATABASE_ID}" ]] && ! cnpg_available \ + && printf '%s' "${body}" | grep -Fq '"provider":"cnpg"'; then + warn "openshell-db ManagedDatabase ${DATABASE_ID} has provider=cnpg, but this cluster has no CNPG operator" + warn "Gateways using it will not reconcile. Run make openshift-down then make openshift-up, or delete that ManagedDatabase and make openshift-seed" + seed_failed=true + fi + if [[ -z "${seed_failed}" && -z "${DATABASE_ID}" ]]; then + info "Creating ManagedDatabase (provider=${db_provider})..." raw="$(api_exec POST /api/hypershell/v1/managed_databases \ - "{\"name\":\"openshell-db\",\"provider\":\"cnpg\"}")" + "{\"name\":\"openshell-db\",\"provider\":\"${db_provider}\"}")" http="$(printf '%s' "${raw}" | tail -1)" body="$(printf '%s' "${raw}" | sed '$d')" if [[ "${http}" != "201" && "${http}" != "200" ]]; then @@ -953,9 +959,9 @@ seed_via_api() { seed_failed=true else DATABASE_ID="$(extract_id "${body}")" - success "ManagedDatabase created: ${DATABASE_ID}" + success "ManagedDatabase created: ${DATABASE_ID} (provider=${db_provider})" fi - else + elif [[ -z "${seed_failed}" ]]; then success "openshell-db ManagedDatabase already exists: ${DATABASE_ID}" fi fi diff --git a/scripts/cluster/lib_test.sh b/scripts/cluster/lib_test.sh index 3755c069..f058d606 100755 --- a/scripts/cluster/lib_test.sh +++ b/scripts/cluster/lib_test.sh @@ -141,6 +141,39 @@ else FAIL=$((FAIL + 1)) echo 'FAIL: OpenShift cluster_up does not honor SKIP_SEED' fi +if grep -A8 'db_provider="cnpg"' "${SCRIPT_DIR}/drivers/openshift.sh" | grep -q 'cnpg_available' \ + && grep -A20 'Creating ManagedDatabase' "${SCRIPT_DIR}/drivers/openshift.sh" | grep -q 'provider='; then + PASS=$((PASS + 1)) +else + FAIL=$((FAIL + 1)) + echo 'FAIL: OpenShift seed still hardcodes ManagedDatabase provider=cnpg' +fi +if grep -A30 '^wait_for_deployments()' "${SCRIPT_DIR}/drivers/openshift.sh" | grep -q 'is_openshift_swapped'; then + FAIL=$((FAIL + 1)) + echo 'FAIL: wait_for_deployments skips swapped components' +else + PASS=$((PASS + 1)) +fi +if grep -A30 '^wait_for_deployments()' "${SCRIPT_DIR}/drivers/openshift.sh" | grep -q 'wait_for_named_rollout hypershell-api-server' \ + && grep -A30 '^wait_for_deployments()' "${SCRIPT_DIR}/drivers/openshift.sh" | grep -q 'wait_for_named_rollout hypershell-controller' \ + && grep -A30 '^wait_for_deployments()' "${SCRIPT_DIR}/drivers/openshift.sh" | grep -q 'wait_for_named_rollout hypershell-web-console'; then + PASS=$((PASS + 1)) +else + FAIL=$((FAIL + 1)) + echo 'FAIL: wait_for_deployments does not wait for platform rollouts' +fi +if grep -qE 'wait_for_oidc_token|wait_for_api_openapi|wait_for_api_healthcheck' "${SCRIPT_DIR}/drivers/openshift.sh"; then + FAIL=$((FAIL + 1)) + echo 'FAIL: wait_for_deployments still probes Routes for HTTP readiness' +else + PASS=$((PASS + 1)) +fi +if grep -A20 'containerPort: 9443' "${SCRIPT_DIR}/../../deploy/base/controller.yaml" | grep -q 'readinessProbe'; then + PASS=$((PASS + 1)) +else + FAIL=$((FAIL + 1)) + echo 'FAIL: control plane Deployment has no readiness probe on the provisioner port' +fi if grep -A15 'Automatic seeding incomplete' "${SCRIPT_DIR}/drivers/openshift.sh" | grep -q 'seed_strict'; then PASS=$((PASS + 1)) else @@ -680,6 +713,12 @@ if bad: FAIL=$((FAIL + 1)) echo 'FAIL: OpenShift overlay dropped HYPERSHELL_SERVICE_ACCOUNT_PROVISIONER_ADDR FQDN' fi + if grep -A1 'name: GATEWAY_API_HTTP_LISTENER_NAME' "${os_out}" | grep -q 'grpc'; then + PASS=$((PASS + 1)) + else + FAIL=$((FAIL + 1)) + echo 'FAIL: OpenShift overlay does not set GATEWAY_API_HTTP_LISTENER_NAME=grpc' + fi else FAIL=$((FAIL + 1)) echo 'FAIL: rewritten overlay render' diff --git a/specs/platform/e2e-testing.spec.md b/specs/platform/e2e-testing.spec.md index 6c6c795e..16ffb841 100644 --- a/specs/platform/e2e-testing.spec.md +++ b/specs/platform/e2e-testing.spec.md @@ -33,7 +33,8 @@ tests/e2e/e2e-openshell.sh (infra-agnostic test logic) │ ├── sources tests/e2e/lib.sh (shared utilities) │ - └── sources driver via E2E_INFRA_DRIVER (required) + └── selects driver by auto-detecting the KUBECONFIG context + (E2E_INFRA_DRIVER overrides detection) │ ├── tests/e2e/drivers/kind.sh (this spec) └── tests/e2e/drivers/openshift.sh (this spec) @@ -134,34 +135,38 @@ deploy/ ### Requirement: Infra Driver Abstraction -The e2e test framework SHALL isolate infrastructure-specific logic into driver scripts located at `tests/e2e/drivers/.sh`. The main test script (`tests/e2e/e2e-openshell.sh`) SHALL remain infrastructure-agnostic and call only driver interface functions for infrastructure-specific operations. The `E2E_INFRA_DRIVER` environment variable SHALL select the driver and MUST be set -- the script SHALL exit with an error if it is unset. +The e2e test framework SHALL isolate infrastructure-specific logic into driver scripts located at `tests/e2e/drivers/.sh`. The main test script (`tests/e2e/e2e-openshell.sh`) SHALL remain infrastructure-agnostic and call only driver interface functions for infrastructure-specific operations. Only `kind` and `openshift` drivers are supported today; additional drivers are follow-up work. -#### Scenario: Kind Driver Selected +The script SHALL auto-detect the driver from the current KUBECONFIG context rather than require the caller to select one: it SHALL query the cluster's API groups and select `openshift` when `route.openshift.io` is present, and `kind` otherwise. The `E2E_INFRA_DRIVER` environment variable SHALL override auto-detection when set. A user SHALL be able to run the suite with no infra-related environment variables and have it correctly select the driver matching the cluster their KUBECONFIG context points at. -- GIVEN `E2E_INFRA_DRIVER=kind` +#### Scenario: Kind Driver Auto-Detected + +- GIVEN the current KUBECONFIG context points at a cluster that does not serve the `route.openshift.io` API group +- AND `E2E_INFRA_DRIVER` is not set - WHEN the e2e test script starts - THEN the `tests/e2e/drivers/kind.sh` driver SHALL be sourced - AND all infrastructure functions SHALL use `kubectl` and Kind-specific discovery (HTTPRoute hostnames, Gateway API status) -#### Scenario: OpenShift Driver Selected +#### Scenario: OpenShift Driver Auto-Detected -- GIVEN `E2E_INFRA_DRIVER=openshift` +- GIVEN the current KUBECONFIG context points at a cluster that serves the `route.openshift.io` API group +- AND `E2E_INFRA_DRIVER` is not set - WHEN the e2e test script starts - THEN the `tests/e2e/drivers/openshift.sh` driver SHALL be sourced - AND all infrastructure functions SHALL use `oc` and OpenShift-specific discovery as `openshift-development.spec.md` defines (Gateway API status and the configured gateway base domain) +#### Scenario: Driver Override + +- GIVEN the current KUBECONFIG context points at a cluster that would auto-detect to a different driver +- AND `E2E_INFRA_DRIVER` is set explicitly +- WHEN the e2e test script starts +- THEN the script SHALL source the driver `E2E_INFRA_DRIVER` names, bypassing auto-detection + #### Scenario: New Driver Extensibility - GIVEN a developer creates `tests/e2e/drivers/eks.sh` implementing all interface functions - WHEN a user runs the e2e tests with `E2E_INFRA_DRIVER=eks` -- THEN the tests SHALL execute against EKS without modifying the main test script - -#### Scenario: Driver Not Set - -- GIVEN `E2E_INFRA_DRIVER` is not set -- WHEN the e2e test script starts -- THEN the script SHALL exit with a non-zero status -- AND print a message listing available drivers from `tests/e2e/drivers/*.sh` +- THEN the tests SHALL execute against EKS without modifying the main test script or the auto-detection logic #### Scenario: Unknown Driver @@ -213,7 +218,7 @@ The OpenShift driver implements the same ten functions with OpenShift constructs ### Requirement: Custom OpenShift Runs -Each target SHALL default `E2E_INFRA_DRIVER` to `kind` and SHALL honor a command-line override. A user SHALL be able to run `make e2e` and `make e2e-performance` **manually** against any OpenShift cluster, so scale and performance testing can target a real OpenShift environment: a user SHALL run `E2E_INFRA_DRIVER=openshift make e2e` or `E2E_INFRA_DRIVER=openshift make e2e-performance` against the cluster their current `oc` context selects. These OpenShift runs SHALL NOT create a cluster and SHALL NOT create a namespace beyond the gateways the suite provisions; the environment is a precondition. +Each target SHALL auto-detect the driver from the current KUBECONFIG context (see [Infra Driver Abstraction](#requirement-infra-driver-abstraction)) and SHALL honor an `E2E_INFRA_DRIVER` command-line override. A user SHALL be able to run `make e2e` and `make e2e-performance` **manually** against any OpenShift cluster, so scale and performance testing can target a real OpenShift environment: a user logged in to an OpenShift cluster via `oc login` SHALL run `make e2e` or `make e2e-performance` and have the suite auto-detect the OpenShift driver, or force it explicitly with `E2E_INFRA_DRIVER=openshift make e2e` / `E2E_INFRA_DRIVER=openshift make e2e-performance`. These OpenShift runs SHALL NOT create a cluster and SHALL NOT create a namespace beyond the gateways the suite provisions; the environment is a precondition. **Preconditions (owned by `openshift-development.spec.md`).** These runs assume HyperShell is already deployed on the cluster through `make openshift-up` (`kustomize build deploy/openshift/` mapped into the current `oc` project, or `OPENSHIFT_NAMESPACE`). That bring-up creates the companion `${OPENSHIFT_NAMESPACE}-keycloak` project, applies Routes for the API, web console, and Keycloak, applies `keycloak-allow-platform` so platform pods can reach JWKS, applies per-environment ClusterRoles and ClusterRoleBindings named `${OPENSHIFT_NAMESPACE}-dev-*`, and applies the privileged SCC RoleBinding `hypershell-sandbox-scc`. The cluster infrastructure bootstrap (shared Gateway, GatewayClass, certificate issuer, wildcard certificate) is in place per `openshift-development.spec.md`. The suite SHALL fail with a clear error, not a broken run, when the API Route or the gateway infrastructure is absent. @@ -228,7 +233,7 @@ Each target SHALL default `E2E_INFRA_DRIVER` to `kind` and SHALL honor a command - GIVEN the OpenShift driver is present at `tests/e2e/drivers/openshift.sh` - AND a user is logged in to an OpenShift cluster with HyperShell deployed via `make openshift-up` - AND the cluster infrastructure bootstrap is in place per `openshift-development.spec.md` -- WHEN the user runs `E2E_INFRA_DRIVER=openshift make e2e` +- WHEN the user runs `make e2e` with no `E2E_INFRA_DRIVER` set, or runs `E2E_INFRA_DRIVER=openshift make e2e` explicitly - THEN the suite SHALL run against that cluster using the OpenShift driver - AND no Kind cluster SHALL be created @@ -709,7 +714,7 @@ deploy/ | Env Var | Default | Description | |---------|---------|-------------| -| `E2E_INFRA_DRIVER` | (required) | Infra driver to use: `kind` or `openshift` | +| `E2E_INFRA_DRIVER` | (auto-detected from KUBECONFIG context) | Infra driver override: `kind` or `openshift` | | `OPENSHIFT_NAMESPACE` | current `oc project` | Platform namespace the OpenShift driver and `make openshift-up` target; Keycloak is `${OPENSHIFT_NAMESPACE}-keycloak` | | `E2E_NAMESPACE` | `openshell-e2e` | Namespace for e2e test resources (gateway deployment) | | `E2E_GATEWAY_NAME` | `e2e-gw` | Gateway name for the e2e test | @@ -802,7 +807,7 @@ The CI e2e workflow SHALL verify web console distributed tracing end to end, sat The performance test measures how the platform behaves when many gateways run at the same time. It provisions a large fleet of gateways on the target cluster in batches. After every batch it runs a fast mini test (the e2e suite in short mode against a canary gateway) and appends a checkpoint record to the results, so a regression is pinned to the scale at which it appears rather than surfacing only at the end. Once the fleet is fully provisioned it runs the full functional e2e suite to confirm the platform still works correctly under that load. A user runs the test with `make e2e-performance`. -The performance test reuses the e2e driver abstraction. It runs against any infrastructure target that supplies a driver. A user selects the target with `E2E_INFRA_DRIVER`, the same as the e2e suite. A user runs the test against Kind for local checks. A user runs the test against any OpenShift cluster for on-demand load tests. +The performance test reuses the e2e driver abstraction. It runs against any infrastructure target that supplies a driver. It auto-detects the target from the current KUBECONFIG context, the same as the e2e suite, with `E2E_INFRA_DRIVER` available as an override. A user runs the test against Kind for local checks. A user runs the test against any OpenShift cluster for on-demand load tests. The performance test does not build images and does not create the cluster. It targets a cluster that already runs. It reuses the resources that `make kind-up` (or the OpenShift deploy) already seeded: one managed cluster, one release, and (when `DATABASE_PROVIDER=cnpg`) one managed database. Each perf gateway create body reuses the cluster and release ids. @@ -814,7 +819,8 @@ tests/e2e/e2e-performance.sh (infra-agnostic performance harness) ├── sources tests/e2e/lib.sh (pass/fail tracking, retry, colors, env defaults) ├── sources tests/e2e/perf-lib.sh (timing, latency percentiles, bounded concurrency) │ - ├── sources driver via E2E_INFRA_DRIVER (required: kind | openshift) + ├── selects driver by auto-detecting the KUBECONFIG context + │ (E2E_INFRA_DRIVER overrides detection; kind | openshift) │ ├── Phase 1 Preflight -- discover API host, OIDC token, cluster/release ids, baseline; │ provision the canary gateway and grant its OIDC role once @@ -831,34 +837,34 @@ The harness holds no infrastructure-specific logic. It calls only driver interfa ### Requirement: Performance Test Entry Point -The system SHALL provide a `make e2e-performance` target. The target SHALL run `tests/e2e/e2e-performance.sh`. The target SHALL default `E2E_INFRA_DRIVER` to `kind` for local use, the same pattern as the `make e2e` target. A user SHALL be able to override the driver on the command line. This lets the same target run against any OpenShift cluster (see [Custom OpenShift Runs](#requirement-custom-openshift-runs)). +The system SHALL provide a `make e2e-performance` target. The target SHALL run `tests/e2e/e2e-performance.sh`. The target SHALL auto-detect the driver from the current KUBECONFIG context, the same pattern as the `make e2e` target. A user SHALL be able to override the driver on the command line with `E2E_INFRA_DRIVER`. This lets the same target run against any OpenShift cluster (see [Custom OpenShift Runs](#requirement-custom-openshift-runs)). #### Scenario: Local Kind Run - GIVEN a developer has a running Kind cluster from `make kind-up` -- WHEN the developer runs `make e2e-performance` -- THEN the harness SHALL run with `E2E_INFRA_DRIVER=kind` +- WHEN the developer runs `make e2e-performance` with no `E2E_INFRA_DRIVER` set +- THEN the harness SHALL auto-detect and run with the `kind` driver - AND it SHALL provision `E2E_PERF_GATEWAY_COUNT` gateways and report performance metrics #### Scenario: OpenShift Run - GIVEN a user is logged in to an OpenShift cluster with HyperShell deployed - AND the `openshift` driver is present at `tests/e2e/drivers/openshift.sh` -- WHEN the user runs `E2E_INFRA_DRIVER=openshift make e2e-performance` +- WHEN the user runs `make e2e-performance` with no `E2E_INFRA_DRIVER` set, or runs `E2E_INFRA_DRIVER=openshift make e2e-performance` explicitly - THEN the harness SHALL run against the OpenShift cluster with no change to the harness code - AND all infrastructure operations SHALL use the OpenShift driver (`oc`, Routes) ### Requirement: Infra-Agnostic Performance Harness -The performance harness (`tests/e2e/e2e-performance.sh`) SHALL be infrastructure-agnostic. It SHALL call only the driver interface functions for infrastructure operations. It SHALL select the driver with `E2E_INFRA_DRIVER`, the same as the e2e suite. It SHALL exit with a non-zero status at startup if `E2E_INFRA_DRIVER` is unset or names a missing driver, and SHALL list the available drivers. It SHALL NOT contain any `kubectl`-only, `oc`-only, or `kind`-only command. +The performance harness (`tests/e2e/e2e-performance.sh`) SHALL be infrastructure-agnostic. It SHALL call only the driver interface functions for infrastructure operations. It SHALL select the driver the same way the e2e suite does: auto-detected from the current KUBECONFIG context, with `E2E_INFRA_DRIVER` as an override. It SHALL exit with a non-zero status at startup if `E2E_INFRA_DRIVER` names a missing driver, and SHALL list the available drivers. It SHALL NOT contain any `kubectl`-only, `oc`-only, or `kind`-only command. The harness SHALL obtain the seeded cluster, release, and managed database ids the same way the e2e suite does: it SHALL query the API through `api_curl` and reuse the shared seeding helpers in `tests/e2e/lib.sh`, never hardcoding ids. When `E2E_SEED_CLUSTER_NAME` / `E2E_SEED_RELEASE_NAME` are set, discovery SHALL select the matching name; when they are unset it SHALL take the first list item (the single-seed Kind/CI layout). On `E2E_INFRA_DRIVER=kind` those names SHALL default to the `make kind-up` seeds (`local-kind`, `dev-release`). Every diagnostic or resource-inspection command SHALL invoke the Kubernetes CLI through `$(get_cli_binary)`, so it resolves to `kubectl` on Kind and `oc` on OpenShift with no change to the harness. The OpenShift driver is specified alongside this contract in `openshift-development.spec.md`; the performance harness uses it for OpenShift runs (see [Scope](#scope)). The harness SHALL contain no infra-specific code: it works with either driver with no change. OpenShift runs are manual and on-demand; the performance test is not wired into CI for any target (see [Design Decisions](#design-decisions)). -#### Scenario: Driver Not Set +#### Scenario: Unknown Driver Override -- GIVEN `E2E_INFRA_DRIVER` is not set +- GIVEN `E2E_INFRA_DRIVER=nonexistent` - WHEN the performance harness starts - THEN it SHALL exit with a non-zero status - AND print the available drivers from `tests/e2e/drivers/*.sh` @@ -1293,7 +1299,7 @@ On failure, the harness SHALL collect diagnostics that explain resource pressure | `E2E_PERF_CSV` | `0` | Set to `1` to also append each run to `/history.csv` | | `E2E_PERF_MIN_SUCCESS_RATE` | (unset) | Optional SLO: min provisioning success rate percent; below this fails the run | | `E2E_PERF_MAX_PROVISION_P99` | (unset) | Optional SLO: max p99 time-to-`Running` seconds; above this fails the run | -| `E2E_INFRA_DRIVER` | (required; `make e2e-performance` defaults to `kind`) | Infra driver: `kind` or `openshift` | +| `E2E_INFRA_DRIVER` | (auto-detected from KUBECONFIG context) | Infra driver override: `kind` or `openshift` | | `E2E_SKIP_CLEANUP` | `0` | Set to `1` to keep the perf fleet after the run | **Capacity note:** a small Kind cluster cannot run hundreds of gateways. Each gateway provisions a deployment, a service, a TLS secret, a certgen job, a per-gateway Keycloak client, and a managed namespace. A run also stands up the canary and the functional gateway, so the cluster carries `E2E_PERF_GATEWAY_COUNT + 2` gateway stacks at peak: the default of 5 means 7 stacks, which fits a typical Kind cluster. Keep the total modest on Kind (roughly `count + 2` at or below 10). Use a larger count on an OpenShift cluster that has spare capacity. The harness reports resource pressure on failure so a user can find the ceiling. @@ -1303,7 +1309,7 @@ On failure, the harness SHALL collect diagnostics that explain resource pressure | Decision | Rationale | |----------|-----------| | Shell-based drivers as starting point | The e2e test is a shell script; shell functions provide the simplest driver abstraction without adding a new language or build step. Each driver is a single file implementing a known function interface. If the test suite grows in complexity -- structured assertions, parallel execution, direct Kubernetes API client usage -- migrating to a Go-based e2e framework (e.g., `go test` with client-go) is a natural follow-up. The driver interface contract is function-shape-agnostic, so the same logical abstraction applies in either language | -| `E2E_INFRA_DRIVER` is required, no auto-detection | Explicit driver selection avoids ambiguity and makes CI invocations self-documenting. Each environment sets the driver it intends to test against | +| `E2E_INFRA_DRIVER` is auto-detected from the KUBECONFIG context, with an explicit override | `route.openshift.io` is a reliable, cheap signal for OpenShift, so a developer running against whichever cluster their context selects does not need to remember to set a flag. CI still sets `E2E_INFRA_DRIVER=kind` explicitly so the invocation stays self-documenting and does not depend on the runner's kubeconfig | | Tests live in `tests/e2e/`, not `components/pr-test/` | A top-level `tests/` tree is the natural home for e2e tests and their drivers. `components/pr-test/` will be deprecated in a follow-up once migration is complete | | Shared test utilities in `tests/e2e/lib.sh` | Pass/fail tracking, color output, and retry helpers are currently inline in `e2e-openshell.sh`. Extracting them into `lib.sh` makes them reusable across future test scripts without duplicating code | | CI pulls Konflux-built images, not rebuild | Images are built by Konflux (the existing build pipeline). The e2e workflow gates on those builds and pulls images by digest, avoiding duplicate builds and ensuring CI tests the exact images that ship. This is expected to cover HYPERSHELL-16 | diff --git a/specs/platform/openshell-gateway-namespace-gc.spec.md b/specs/platform/openshell-gateway-namespace-gc.spec.md index 2df24ebf..ab02ab0d 100644 --- a/specs/platform/openshell-gateway-namespace-gc.spec.md +++ b/specs/platform/openshell-gateway-namespace-gc.spec.md @@ -56,9 +56,9 @@ Namespace creation and provisioning mechanics are defined in names match the gateway prefix (`openshell-`) and excludes ManagedDatabase namespaces (`openshell-db-`). A namespace owned by a different instance is never listed, annotated, or reaped. A gateway namespace that carries both - management labels but no instance label is a pre-label leftover: each GC sweep - claims it for this instance (stamps `hypershell.redhat.io/instance`) and then - evaluates it as any other owned namespace. ManagedDatabase namespaces and + management labels but no instance label is unlabeled leftover. Periodic GC + SHALL leave it unlabeled. An operator labels it for the owning instance when + they want that instance's sweep to consider it. ManagedDatabase namespaces and namespaces already labeled for another instance are never claimed. A Gateway pointed at a pre-existing or shared namespace can never cause that namespace to be reaped. @@ -157,16 +157,10 @@ same cluster is never observed as an orphan of this instance. If `HYPERSHELL_NAMESPACE` is empty, the reconciler SHALL abort the sweep rather than list by the generic management labels alone. -Before that instance-scoped list, each sweep SHALL claim unlabeled legacy -gateway namespaces: those that carry both management labels, have no -`hypershell.redhat.io/instance` key, and match the gateway prefix (not -`openshell-db-`). Claiming SHALL stamp this instance's identity and SHALL -NOT overwrite a label that already identifies a different instance. Unlabeled -ManagedDatabase namespaces SHALL NOT be claimed. After claiming, the sweep SHALL -evaluate those namespaces as owned by this instance (live Gateways are retained; -orphans start or continue the grace timer and are reaped after it elapses). If -claiming fails, the reconciler SHALL still sweep already-labeled namespaces. -The sweep interval defaults to +The sweep SHALL NOT stamp `hypershell.redhat.io/instance` onto unlabeled +namespaces. Unlabeled leftovers stay unlabeled until an operator assigns the +owning instance. Live Gateways this instance reconciles still receive the +instance label through `EnsureManagedNamespace`. The sweep interval defaults to 5 minutes and the grace period defaults to 10 minutes. Garbage collection SHALL be enabled by default and configurable without code changes via environment variables: @@ -197,25 +191,24 @@ database prefix, and carrying this instance's identity label). - THEN it SHALL NOT list, annotate, or delete instance A's namespaces - AND it SHALL NOT record them as orphaned of instance B -#### Scenario: Unlabeled legacy orphan is claimed then swept +#### Scenario: Unlabeled leftover is left for an operator - GIVEN a namespace that carries the two management labels but not `hypershell.redhat.io/instance` - AND its name is gateway-prefixed (`openshell-`, not `openshell-db-`) - AND no live Gateway in this instance's API server maps to it - WHEN this instance's garbage-collection reconciler sweeps -- THEN it SHALL stamp `hypershell.redhat.io/instance` for this instance -- AND it SHALL treat the namespace as an orphan of this instance (stamp - `gc-eligible-since`, and delete after the grace period) +- THEN it SHALL NOT stamp `hypershell.redhat.io/instance` +- AND it SHALL NOT stamp `gc-eligible-since` or delete that namespace -#### Scenario: Live unlabeled gateway namespace is labeled but not orphaned +#### Scenario: Live unlabeled gateway namespace is labeled on reconcile, not by GC - GIVEN a namespace that carries the two management labels but not `hypershell.redhat.io/instance` - AND a live Gateway in this instance's API server maps to it -- WHEN this instance's garbage-collection reconciler sweeps +- WHEN the control plane reconciles that Gateway - THEN it SHALL stamp `hypershell.redhat.io/instance` for this instance -- AND it SHALL NOT stamp `gc-eligible-since` or delete that namespace +- AND the garbage-collection sweep SHALL NOT stamp that label on its own #### Scenario: Unlabeled ManagedDatabase namespace is not claimed @@ -242,11 +235,10 @@ database prefix, and carrying this instance's identity label). - GIVEN the control plane was down when a Gateway is deleted, so the namespace was never reaped by the delete path -- AND the namespace is either labeled for this instance or is an unlabeled - legacy namespace carrying both management labels +- AND the namespace already carries this instance's identity label - WHEN the control plane restarts and the garbage-collection reconciler sweeps -- THEN it SHALL observe the namespace as orphaned (claiming an unlabeled - leftover first) and, once the grace period has elapsed, delete it +- THEN it SHALL observe the namespace as orphaned and, once the grace period has + elapsed, delete it ### Requirement: Stamp This Instance's Identity on Namespaces It Manages @@ -257,12 +249,11 @@ two management labels. If the namespace already carries a different instance identity, the control plane SHALL NOT adopt, relabel, or delete it. Reconciling a live Gateway SHALL add the instance label to a legacy unlabeled namespace this instance is actively managing, so a later orphan can be reaped by this instance's -periodic GC. Each GC sweep SHALL also stamp this instance's identity onto -unlabeled legacy gateway namespaces (both management labels, no instance label, -gateway prefix) so a missed-delete orphan that never hits the live-reconcile -path becomes eligible for the orphan sweep. An empty `HYPERSHELL_NAMESPACE` -SHALL NOT create or relabel a managed namespace. Two controllers on the same -cluster SHALL NOT share an instance identity. +periodic GC. Periodic GC SHALL NOT stamp instance labels onto unlabeled +namespaces. An operator who wants an unlabeled leftover swept assigns +`hypershell.redhat.io/instance` to the owning control-plane namespace. An empty +`HYPERSHELL_NAMESPACE` SHALL NOT create or relabel a managed namespace. Two +controllers on the same cluster SHALL NOT share an instance identity. #### Scenario: Created gateway namespace is labeled for this instance diff --git a/specs/platform/openshift-development.spec.md b/specs/platform/openshift-development.spec.md index 610e097d..31f53d3d 100644 --- a/specs/platform/openshift-development.spec.md +++ b/specs/platform/openshift-development.spec.md @@ -89,7 +89,8 @@ Makefile (single entry point) │ └── e2e tests (infra-agnostic, unchanged contract) │ - └── selects driver via E2E_INFRA_DRIVER + └── selects driver by auto-detecting the KUBECONFIG context + (E2E_INFRA_DRIVER overrides detection) ├── tests/e2e/drivers/kind.sh └── tests/e2e/drivers/openshift.sh (this spec) ``` @@ -129,11 +130,14 @@ OpenShift lifecycle driver and the OpenShift e2e driver The lifecycle driver and the e2e driver for a target SHALL select the same infrastructure. The lifecycle driver comes from the make target name. The e2e -driver comes from `E2E_INFRA_DRIVER`, which the test entry points require (see -`e2e-testing.spec.md`). The test entry points (`make e2e`, `make e2e-performance`) -are each one infrastructure-agnostic target, so they need an explicit selector. The -lifecycle targets do not need a selector, because the target name already fixes the -infrastructure. +driver auto-detects from the current KUBECONFIG context, which the test entry +points require to already point at the target infrastructure (see +`e2e-testing.spec.md`); `E2E_INFRA_DRIVER` remains available to override +detection. The test entry points (`make e2e`, `make e2e-performance`) are each +one infrastructure-agnostic target, so a developer who has run `make +openshift-up` and left their `oc` context pointed at that cluster gets the +OpenShift e2e driver with no extra selector. The lifecycle targets do not need +a selector either, because the target name already fixes the infrastructure. #### Scenario: Default driver preserves Kind behavior @@ -190,7 +194,15 @@ Like `make kind-up`, `make openshift-up` SHALL seed the domain resources a developer needs for a working gateway -- a ManagedCluster, a GatewayRelease, a ManagedDatabase, and a Gateway -- with the OpenShift Route and OIDC values for the environment, so that one command produces a working gateway and -the OpenShift workflow matches the Kind workflow. +the OpenShift workflow matches the Kind workflow. When the CloudNativePG operator +is not on the cluster, `make openshift-up` already falls back to the bundled +PostgreSQL Deployment for the API server; seeding SHALL create the ManagedDatabase +with `provider=deployment` in that case, not `provider=cnpg`. When CNPG is +present, seeding MAY use `provider=cnpg`. The OpenShift overlay SHALL set +`GATEWAY_API_HTTP_LISTENER_NAME=grpc` so console HTTPRoutes attach to the shared +Gateway listener of that name. The default `https` sectionName SHALL NOT be used +on this overlay: the shared Gateway has no `https` listener, and that mismatch +reports `NoMatchingParent` and does not self-heal. The `make openshift-down` command SHALL delete the applied manifests and SHALL remove every project in the environment namespace group: the platform project @@ -213,6 +225,19 @@ component swap state, the same categories that `make kind-status` reports. The command names SHALL mirror the Kind command names by replacing the `kind` prefix with `openshift`. +Like `make kind-up`, `make openshift-up` SHALL wait until the stack is ready +before it prints the running banner or seeds. The wait SHALL cover Keycloak, +PostgreSQL when it is deployed as a Deployment, the API server, the control +plane, and the web console, including after Route-derived environment updates +and including a swapped working-tree image. The wait SHALL use +`oc rollout status`, not `oc wait --for=condition=available`, so a Deployment +that stays Available during a rolling update cannot report ready while a new +ReplicaSet is still in flight. The control plane Deployment SHALL expose a TCP +readiness probe on the service-account provisioner port so rollout is not +complete until that port is listening. The wait SHALL NOT probe OpenShift +Routes for `/healthcheck`, `/openapi`, or an OIDC token; seeding already retries +those from the developer machine. + #### Scenario: Deploy the full stack to OpenShift - GIVEN a developer has a kubeconfig context for an OpenShift cluster @@ -225,6 +250,15 @@ prefix with `openshift`. - AND the scripts report the API Route, the web-console Route, and the Keycloak Route when the deployment is ready +#### Scenario: Wait until the stack can serve + +- GIVEN overlay apply, swap restore, and Route-derived environment updates have + triggered rollouts +- WHEN the developer runs `make openshift-up` +- THEN the command does not print the running banner until `oc rollout status` + succeeds for Keycloak, the API server, the control plane, and the web console +- AND the command does not skip that wait for a swapped component + #### Scenario: Console login and API seeding use the OpenShift Routes The API server image does not include `curl`. Token grants, Keycloak Admin @@ -569,9 +603,12 @@ cluster-wildcard certificate, the suite SHALL rely on the system trust store. Wh the shared Gateway serves a private CA, the driver SHALL extract that CA and point `SSL_CERT_FILE` at it. The suite SHALL NOT set `OPENSHELL_GATEWAY_INSECURE`. -The OpenShift e2e suite SHALL run with `E2E_INFRA_DRIVER=openshift bash -tests/e2e/e2e-openshell.sh`, and SHALL exercise the same test areas that the Kind -suite exercises, so that a single suite validates both infrastructure targets. +The OpenShift e2e suite SHALL run with `bash tests/e2e/e2e-openshell.sh` against +a KUBECONFIG context pointed at the OpenShift cluster -- the suite auto-detects +the OpenShift driver from that context, or a caller MAY force it explicitly +with `E2E_INFRA_DRIVER=openshift bash tests/e2e/e2e-openshell.sh` -- and SHALL +exercise the same test areas that the Kind suite exercises, so that a single +suite validates both infrastructure targets. #### Scenario: Discover the API host on OpenShift diff --git a/tests/e2e/e2e-openshell.sh b/tests/e2e/e2e-openshell.sh index f1b82c72..ecaf4c8c 100755 --- a/tests/e2e/e2e-openshell.sh +++ b/tests/e2e/e2e-openshell.sh @@ -5,17 +5,19 @@ # Proves the full path: HyperShell API -> control plane -> gateway provisioning # -> openshell CLI -> sandbox pod creation + interaction. # -# The E2E_INFRA_DRIVER environment variable selects the infrastructure driver. -# Each driver (tests/e2e/drivers/.sh) implements a fixed set of -# functions that abstract infrastructure-specific operations. +# The infrastructure driver is auto-detected from the current KUBECONFIG +# context: a cluster that serves the route.openshift.io API group is +# OpenShift, otherwise Kind is assumed. Set E2E_INFRA_DRIVER to override +# detection. Each driver (tests/e2e/drivers/.sh) implements a fixed +# set of functions that abstract infrastructure-specific operations. # # Usage: -# E2E_INFRA_DRIVER=kind bash tests/e2e/e2e-openshell.sh +# bash tests/e2e/e2e-openshell.sh # OPENSHIFT_NAMESPACE=my-env E2E_INFRA_DRIVER=openshift \ -# bash tests/e2e/e2e-openshell.sh +# bash tests/e2e/e2e-openshell.sh # override detection # # Environment variables: -# E2E_INFRA_DRIVER (required) Infra driver: kind or openshift +# E2E_INFRA_DRIVER Infra driver override: kind, openshift (default: auto-detected) # E2E_NAMESPACE Namespace for e2e resources (default: openshell-e2e) # E2E_GATEWAY_NAME Gateway name (default: e2e-gw) # E2E_MODE Run depth: long (default, every step) or short (essential steps) @@ -46,8 +48,20 @@ DB_PROVIDER="${DATABASE_PROVIDER:-deployment}" e2e_validate_mode +# Detects OpenShift by checking whether the current KUBECONFIG context serves +# the route.openshift.io API group, an API only OpenShift clusters expose. +# Any other cluster is assumed to be Kind. +detect_infra_driver() { + if kubectl api-versions 2>/dev/null | grep -q '^route\.openshift\.io/'; then + echo "openshift" + else + echo "kind" + fi +} + if [[ -z "${E2E_INFRA_DRIVER:-}" ]]; then - e2e_die_unknown_driver "E2E_INFRA_DRIVER is not set." + E2E_INFRA_DRIVER="$(detect_infra_driver)" + dim " Detected infra driver: ${E2E_INFRA_DRIVER} (from KUBECONFIG context; set E2E_INFRA_DRIVER to override)" fi DRIVER_FILE="${SCRIPT_DIR}/drivers/${E2E_INFRA_DRIVER}.sh" From d566efb2bee9e8c07ed335394eced80b0e1aae42 Mon Sep 17 00:00:00 2001 From: Kyle Squizzato Date: Wed, 2 Sep 2026 19:34:11 -0700 Subject: [PATCH 10/14] feat(e2e-openshift): Publish swaps to quay.io, further bug fixes for gc Assisted-by: Claude Sonnet 5 Signed-off-by: Kyle Squizzato --- DEVELOPMENT.md | 53 +++-- Makefile | 6 +- components/api-server/Dockerfile | 3 +- components/control-plane/Dockerfile | 3 +- .../cmd/hypershell-controller/main.go | 98 +++++++-- .../internal/gateway/namespace.go | 47 ---- .../internal/gateway/namespace_test.go | 87 -------- .../internal/reconciler/namespace.go | 28 +-- .../internal/reconciler/namespace_test.go | 81 ++----- deploy/base/api-server.yaml | 13 +- deploy/openshift/kustomization.yaml | 10 +- scripts/cluster/drivers/openshift.sh | 101 +++++---- scripts/cluster/lib.sh | 206 ++++++++++++++++++ scripts/cluster/lib_test.sh | 158 +++++++++++++- scripts/kind/up.sh | 9 +- specs/platform/local-development.spec.md | 13 +- .../openshell-gateway-namespace-gc.spec.md | 13 +- specs/platform/openshift-development.spec.md | 92 +++++++- 18 files changed, 686 insertions(+), 335 deletions(-) diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md index 92beda63..2f8ca2f1 100644 --- a/DEVELOPMENT.md +++ b/DEVELOPMENT.md @@ -300,11 +300,13 @@ curl -s -H "Authorization: Bearer ${TOKEN}" \ If your baseline images live in a private registry, provide a pull secret: ```bash -KIND_PULL_SECRET=/path/to/pull-secret.yaml make kind-up +PULL_SECRET=/path/to/pull-secret.yaml make kind-up ``` -The YAML file is applied into the target namespace with `kubectl apply`. It -should contain a `kubernetes.io/dockerconfigjson` Secret. +`KIND_PULL_SECRET` is still accepted as an alias. The YAML file is applied into +the target namespace with `kubectl apply`. It should contain a +`kubernetes.io/dockerconfigjson` Secret. OpenShift component swaps use the +same file to log the container engine into `SWAP_REGISTRY`. ### Offline development @@ -329,7 +331,8 @@ build time, so no external dependency checkout is needed. | `KIND_HOST_MOUNT_PATH` | Repository root | Host directory mounted into Kind nodes | | `KIND_KEYCLOAK_URL` | (unset) | External Keycloak URL; skips local deploy | | `KEYCLOAK_OIDC_ISSUER` | `https://keycloak.hypershell.localhost/realms/hypershell` | OIDC issuer URL | -| `KIND_PULL_SECRET` | (unset) | Path to pull secret YAML for private registries | +| `PULL_SECRET` | (unset) | Path to a `kubernetes.io/dockerconfigjson` Secret YAML for private registries. Used by Kind `kind-up` and OpenShift swaps. | +| `KIND_PULL_SECRET` | (unset) | Alias for `PULL_SECRET`. | | `IMAGE_REGISTRY` | `quay.io/redhat-services-prod/hcm-eng-prod-tenant/hypershell-main` | Container registry for baseline images | | `IMAGE_TAG` | `latest` | Image tag for baseline images | | `LOCAL_IMAGES` | (unset) | Set to `true` to build baseline images from the working tree | @@ -383,7 +386,7 @@ command stops with an error. | `make openshift-teardown` | Same as `openshift-down`. There is no OpenShift cluster to destroy. | | `make openshift-status` | Show namespaces, pods, Routes, the shared Gateway, and swap state. | | `make openshift-seed` | Re-run ManagedCluster, GatewayRelease, ManagedDatabase, and Gateway seeding via API and Keycloak Routes from this machine. `openshift-up` already seeds unless `SKIP_SEED=true`. | -| `make openshift-api-server-up` | Build, push an immutable image to the internal registry, and swap the API server. | +| `make openshift-api-server-up` | Build, push an immutable image to `SWAP_REGISTRY`, and point the API server Deployment at that ref. Requires `SWAP_REGISTRY`. | | `make openshift-api-server-down` | Revert the API server to the baseline registry image. | | `make openshift-control-plane-up` | Build, push, and swap the control plane. | | `make openshift-control-plane-down` | Revert the control plane to the baseline registry image. | @@ -446,12 +449,18 @@ make openshift-web-console-up ``` Each swap builds from the working tree, pushes an immutable image (commit + -namespace) to the OpenShift internal registry, and rolls out that exact -identity. Matching `-down` targets revert to the baseline registry image. -`make openshift-status` reports which components run a working-tree build -and the exact image each one uses. Swap state is tracked per namespace in -`.openshift-swaps/` (gitignored). A subsequent `make openshift-up` preserves -active swaps. +namespace tag) to `SWAP_REGISTRY`, and updates the Deployment image refs to +the registry digest from that push (not the local image digest). Images are +built for the OpenShift node architecture (`SWAP_PLATFORM`, or detected from +the cluster), not the laptop architecture. `SWAP_REGISTRY` is the org prefix only (`quay.io/`). Repo +names default to `hypershell-api-server`, `hypershell-controller`, and +`hypershell-web-console`; set `SWAP_REPOSITORY` to override the repo for the +current swap. `SWAP_REGISTRY` is required; swaps do not use `IMAGE_REGISTRY`. +Matching `-down` targets revert to the baseline registry image. Registry +login uses `PULL_SECRET` (`KIND_PULL_SECRET` still works). `make openshift-status` +reports which components run a working-tree build and the exact image each +one uses. Swap state is tracked per namespace in `.openshift-swaps/` +(gitignored). A subsequent `make openshift-up` preserves active swaps. ### OpenShift environment variables @@ -460,7 +469,10 @@ active swaps. | `OPENSHIFT_NAMESPACE` | `oc project -q` | Override for the platform namespace. Unset, the current oc project is used. Max 54 chars; Keycloak lands in `${name}-keycloak`. | | `GATEWAY_API_GATEWAY_NAME` | `openshell-grpc-gateway` | Pre-existing shared Gateway name | | `GATEWAY_API_GATEWAY_NAMESPACE` | `openshift-ingress` | Namespace of the shared Gateway | -| `OPENSHIFT_IMAGE_REGISTRY` | `oc registry info` | Registry used to push swapped images | +| `SWAP_REGISTRY` | (required for swaps) | Registry org prefix only, for example `quay.io/`. Swaps push `${SWAP_REGISTRY}/hypershell-api-server` (and `hypershell-controller`, `hypershell-web-console`). Must be laptop-reachable; the OpenShift internal registry is refused. `IMAGE_REGISTRY` is not used. | +| `SWAP_REPOSITORY` | (component default) | Optional repo name override for the current swap. Default is `hypershell-api-server`, `hypershell-controller`, or `hypershell-web-console`. | +| `SWAP_PLATFORM` | (cluster nodes) | Target platform for swap images, `linux/amd64` or `linux/arm64`. Unset, the scripts use the architecture of the cluster nodes. | +| `PULL_SECRET` | (unset) | Path to a `kubernetes.io/dockerconfigjson` Secret YAML used to log the container engine into `SWAP_REGISTRY`. `KIND_PULL_SECRET` is still accepted. | | `IMAGE_REGISTRY` | `quay.io/redhat-services-prod/hcm-eng-prod-tenant/hypershell-main` | Container registry for baseline images | | `IMAGE_TAG` | `latest` | Image tag for baseline images | | `CONTAINER_ENGINE` | Auto-detected | `podman` or `docker` | @@ -643,5 +655,20 @@ offline mode or configure a pull secret: LOCAL_IMAGES=true make kind-up # Or: provide registry credentials -KIND_PULL_SECRET=/path/to/pull-secret.yaml make kind-up +PULL_SECRET=/path/to/pull-secret.yaml make kind-up ``` + +### OpenShift swap cannot push + +Swaps build on the laptop and push to `SWAP_REGISTRY`, then update the +Deployment image refs. They do not use `IMAGE_REGISTRY` or the OpenShift +internal registry. If `SWAP_REGISTRY` is unset, the swap stops with an error. + +```bash +PULL_SECRET=/path/to/pull-secret.yaml SWAP_REGISTRY=quay.io/ make openshift-api-server-up +``` + +That logs the container engine in with the pull secret and pushes +`quay.io//hypershell-api-server:-`. Override the repo +name with `SWAP_REPOSITORY` when it is not the default. `KIND_PULL_SECRET` is +still accepted if `PULL_SECRET` is unset. diff --git a/Makefile b/Makefile index a917c9c1..491d914f 100644 --- a/Makefile +++ b/Makefile @@ -33,6 +33,7 @@ KIND_HOT_RELOAD?=true KIND_HOST_MOUNT_PATH?=$(shell git rev-parse --show-toplevel 2>/dev/null || pwd) KIND_KEYCLOAK_URL?= LOCAL_IMAGES?= +PULL_SECRET?= KIND_PULL_SECRET?= # Prerequisite versions @@ -309,7 +310,7 @@ test-all: install-js export CONTAINER_ENGINE KIND_CLUSTER_NAME KIND_NAMESPACE export KIND_HOT_RELOAD KIND_HOST_MOUNT_PATH KIND_KEYCLOAK_URL LOCAL_IMAGES BUILD_SOURCE -export KIND_PULL_SECRET +export KIND_PULL_SECRET PULL_SECRET export GATEWAY_API_VERSION KIND_VERSION CLOUD_PROVIDER_KIND_REPO CLOUD_PROVIDER_KIND_REF CLOUD_PROVIDER_KIND_BRANCH CERT_MANAGER_VERSION CNPG_VERSION AGENT_SANDBOX_VERSION export HYPERSHELL_DATABASE_IMAGE export IMAGE_REGISTRY IMAGE_TAG KIND_CONFIG @@ -319,7 +320,7 @@ export api_server_local control_plane_local web_console_local export build_version build_time export API_HOSTNAME CONSOLE_HOSTNAME HEALTH_HOSTNAME KEYCLOAK_HOSTNAME METRICS_HOSTNAME KEYCLOAK_OIDC_ISSUER export KIND_DNS_PORT -export OPENSHIFT_NAMESPACE OPENSHIFT_IMAGE_REGISTRY SKIP_SEED SEED_STRICT +export OPENSHIFT_NAMESPACE SWAP_REGISTRY SWAP_REPOSITORY SWAP_PLATFORM SWAP_ARCH PULL_SECRET SKIP_SEED SEED_STRICT export GATEWAY_API_GATEWAY_NAME GATEWAY_API_GATEWAY_NAMESPACE GATEWAY_IMAGE # Build cloud-provider-kind from a fork that adds BackendTLSPolicy support @@ -377,6 +378,7 @@ kind-env: @echo "export KIND_HOST_MOUNT_PATH=$(KIND_HOST_MOUNT_PATH)" @echo "export KIND_KEYCLOAK_URL=$(KIND_KEYCLOAK_URL)" @echo "export LOCAL_IMAGES=$(LOCAL_IMAGES)" + @echo "export PULL_SECRET=$(PULL_SECRET)" @echo "export KIND_PULL_SECRET=$(KIND_PULL_SECRET)" @echo "export KIND_DB_IMAGE=$(KIND_DB_IMAGE)" @echo "export GATEWAY_API_VERSION=$(GATEWAY_API_VERSION)" diff --git a/components/api-server/Dockerfile b/components/api-server/Dockerfile index 624a6945..736a320e 100644 --- a/components/api-server/Dockerfile +++ b/components/api-server/Dockerfile @@ -13,7 +13,8 @@ COPY openapi/ openapi/ ARG GIT_VERSION=unknown ARG BUILD_TIME=unknown -RUN CGO_ENABLED=0 go build -mod=mod -ldflags="-s -w \ +ARG TARGETARCH +RUN CGO_ENABLED=0 GOOS=linux GOARCH="${TARGETARCH:-$(go env GOARCH)}" go build -mod=mod -ldflags="-s -w \ -X github.com/openshift-online/hypershell/components/api-server/pkg/api.Version=${GIT_VERSION} \ -X 'github.com/openshift-online/hypershell/components/api-server/pkg/api.BuildTime=${BUILD_TIME}'" \ -o hypershell ./cmd/hypershell diff --git a/components/control-plane/Dockerfile b/components/control-plane/Dockerfile index aa43f90d..210d3e45 100644 --- a/components/control-plane/Dockerfile +++ b/components/control-plane/Dockerfile @@ -16,7 +16,8 @@ COPY components/api-server/proto/ components/api-server/proto/ COPY components/control-plane/cmd/ components/control-plane/cmd/ COPY components/control-plane/internal/ components/control-plane/internal/ -RUN cd components/control-plane && CGO_ENABLED=0 go build -mod=mod -ldflags="-s -w" -o /workspace/hypershell-controller ./cmd/hypershell-controller +ARG TARGETARCH +RUN cd components/control-plane && CGO_ENABLED=0 GOOS=linux GOARCH="${TARGETARCH:-$(go env GOARCH)}" go build -mod=mod -ldflags="-s -w" -o /workspace/hypershell-controller ./cmd/hypershell-controller FROM registry.access.redhat.com/hi/static:1787099997@sha256:f4d5109b57cf7eab0a7adc566f2d78f80fa0c5ec9ccab698c9fb8eb448db6071 diff --git a/components/control-plane/cmd/hypershell-controller/main.go b/components/control-plane/cmd/hypershell-controller/main.go index 2046e0d0..2af9a20a 100644 --- a/components/control-plane/cmd/hypershell-controller/main.go +++ b/components/control-plane/cmd/hypershell-controller/main.go @@ -2,11 +2,12 @@ package main import ( "context" - "fmt" "log" "os" "os/signal" + "sync" "syscall" + "time" "google.golang.org/grpc" "google.golang.org/grpc/credentials/insecure" @@ -37,6 +38,39 @@ func managedDatabaseWatchEligible(clientset *kubernetes.Clientset, dynamicClient return clientset != nil && dynamicClient != nil } +// supervisedRestartCap bounds the backoff between restarts of a supervised +// background component, matching the cap watchLoop uses for stream +// reconnects. +const supervisedRestartCap = 30 * time.Second + +// runSupervised runs fn in a loop so a failure in one background component +// (a watch stream, a reconciler's Run loop, the service-account provisioner) +// cannot take down the others by exiting the whole process. Every fn here is +// expected to block until ctx is done and return ctx.Err() at that point, +// exactly as watchLoop and the reconciler Run methods already do; a return +// before then is treated as a failure of that component alone; it is retried +// with the same capped exponential backoff watchLoop uses for stream +// reconnects rather than propagated to the process. +func runSupervised(ctx context.Context, name string, fn func(context.Context) error) { + backoff := time.Second + for { + err := fn(ctx) + if err == nil || ctx.Err() != nil { + return + } + log.Printf("WARN %s exited with error, restarting in %s: %v", name, backoff, err) + select { + case <-ctx.Done(): + return + case <-time.After(backoff): + } + backoff *= 2 + if backoff > supervisedRestartCap { + backoff = supervisedRestartCap + } + } +} + func main() { cfg, err := config.Load() if err != nil { @@ -235,32 +269,54 @@ func main() { if roleBindingReconciler != nil { watchCount++ } - // +4 for the continuous gateway health, namespace GC, sandbox-count, and - // internal service-account provisioner goroutines. - errCh := make(chan error, watchCount+4) + + // Each background component below runs under runSupervised on its own + // goroutine: a failure in one (a dropped watch stream, a reconciler's Run + // loop returning, the service-account provisioner's listener dying) is + // logged and retried in place, and never takes the others down with it. + // wg lets main block until every component has actually observed ctx + // cancellation and returned, instead of exiting out from under them. + var wg sync.WaitGroup + supervise := func(name string, fn func(context.Context) error) { + wg.Go(func() { + runSupervised(ctx, name, fn) + }) + } if cfg.ServiceAccountProvisionerAddress != "" { provisionerServer := serviceaccountprovisioner.NewServer(serviceAccountProvider) transportConfig := serviceaccountprovisioner.TransportConfig{ Address: cfg.ServiceAccountProvisionerAddress, } - go func() { - errCh <- serviceaccountprovisioner.ListenAndServe(ctx, transportConfig, provisionerServer) - }() + supervise("service-account provisioner", func(ctx context.Context) error { + return serviceaccountprovisioner.ListenAndServe(ctx, transportConfig, provisionerServer) + }) log.Printf("INFO service-account provisioner launched on %s (in-cluster, NetworkPolicy-restricted)", cfg.ServiceAccountProvisionerAddress) } else { log.Printf("INFO service-account provisioner disabled") } - go func() { errCh <- watcher.WatchManagedClusters(ctx, conn, clusterReconciler) }() + supervise("ManagedCluster watch", func(ctx context.Context) error { + return watcher.WatchManagedClusters(ctx, conn, clusterReconciler) + }) if databaseReconciler != nil { - go func() { errCh <- watcher.WatchManagedDatabases(ctx, conn, databaseReconciler) }() + supervise("ManagedDatabase watch", func(ctx context.Context) error { + return watcher.WatchManagedDatabases(ctx, conn, databaseReconciler) + }) } - go func() { errCh <- watcher.WatchGatewayReleases(ctx, conn, releaseReconciler) }() - go func() { errCh <- watcher.WatchGateways(ctx, conn, gatewayReconciler) }() - go func() { errCh <- watcher.WatchGatewayNetworks(ctx, conn, networkReconciler) }() + supervise("GatewayRelease watch", func(ctx context.Context) error { + return watcher.WatchGatewayReleases(ctx, conn, releaseReconciler) + }) + supervise("Gateway watch", func(ctx context.Context) error { + return watcher.WatchGateways(ctx, conn, gatewayReconciler) + }) + supervise("GatewayNetwork watch", func(ctx context.Context) error { + return watcher.WatchGatewayNetworks(ctx, conn, networkReconciler) + }) if roleBindingReconciler != nil { - go func() { errCh <- watcher.WatchRoleBindings(ctx, conn, roleBindingReconciler) }() + supervise("RoleBinding watch", func(ctx context.Context) error { + return watcher.WatchRoleBindings(ctx, conn, roleBindingReconciler) + }) } log.Printf("INFO all %d watch streams launched", watchCount) @@ -270,7 +326,7 @@ func main() { // It requires an in-cluster Kubernetes client to observe Deployments. if clientset != nil { healthReconciler := reconciler.NewGatewayHealthReconciler(clientset, dynamicClient, conn, exposurePort, keycloakConfig) - go func() { errCh <- healthReconciler.Run(ctx) }() + supervise("gateway health reconciler", healthReconciler.Run) log.Printf("INFO gateway health reconciler launched") } else { log.Printf("WARN no kubernetes client available, gateway health reconciliation disabled") @@ -282,7 +338,7 @@ func main() { // in-cluster Kubernetes client to watch pods. if clientset != nil { sandboxCountReconciler := reconciler.NewSandboxCountReconciler(clientset, conn, 0) - go func() { errCh <- sandboxCountReconciler.Run(ctx) }() + supervise("sandbox count reconciler", sandboxCountReconciler.Run) log.Printf("INFO sandbox count reconciler launched") } else { log.Printf("WARN no kubernetes client available, sandbox count reconciliation disabled") @@ -296,20 +352,16 @@ func main() { gcReconciler := reconciler.NewNamespaceGCReconciler( clientset, conn, cfg.NamespaceGCInterval, cfg.NamespaceGCGracePeriod, cfg.Namespace, ) - go func() { errCh <- gcReconciler.Run(ctx) }() + supervise("namespace GC reconciler", gcReconciler.Run) log.Printf("INFO namespace GC reconciler launched (interval=%s grace=%s)", cfg.NamespaceGCInterval, cfg.NamespaceGCGracePeriod) } else if clientset != nil { log.Printf("INFO namespace GC reconciler disabled (GATEWAY_NAMESPACE_GC_ENABLED=false)") } - watchErr := <-errCh - cancel() - - if watchErr != nil && watchErr != context.Canceled { - fmt.Fprintf(os.Stderr, "fatal: %v\n", watchErr) - os.Exit(1) - } + <-ctx.Done() + log.Printf("INFO shutdown signal received, waiting for background components to stop") + wg.Wait() log.Printf("INFO hypershell-controller stopped") } diff --git a/components/control-plane/internal/gateway/namespace.go b/components/control-plane/internal/gateway/namespace.go index f97cebd5..36819f22 100644 --- a/components/control-plane/internal/gateway/namespace.go +++ b/components/control-plane/internal/gateway/namespace.go @@ -2,7 +2,6 @@ package gateway import ( "context" - "errors" "fmt" "log" "strings" @@ -66,52 +65,6 @@ func ManagedNamespaceSelector(instance string) (string, error) { ManagedLabel, ManagedLabelValue, ManagedByLabel, ManagedByValue, InstanceLabel, instance), nil } -// LegacyUnlabeledSelector selects HyperShell-managed namespaces that predate the -// instance label: both management labels, and no hypershell.redhat.io/instance -// key. Periodic GC uses this to claim leftover gateway namespaces so a -// missed-delete orphan is not invisible forever. Namespaces already labeled for -// any instance are excluded. -func LegacyUnlabeledSelector() string { - return fmt.Sprintf("%s=%s,%s=%s,!%s", - ManagedLabel, ManagedLabelValue, ManagedByLabel, ManagedByValue, InstanceLabel) -} - -// BackfillInstanceLabels stamps this instance's identity onto unlabeled legacy -// gateway namespaces (both management labels, no instance label, openshell-* -// and not openshell-db-*). It never creates namespaces, never overwrites a -// different instance label, and never claims ManagedDatabase namespaces. An -// empty instance is a configuration error: unlabeled namespaces must not be -// claimed without an identity. -func BackfillInstanceLabels(ctx context.Context, client kubernetes.Interface, instance string) error { - if instance == "" { - return fmt.Errorf("instance identity is empty; refusing to backfill namespace labels") - } - list, err := client.CoreV1().Namespaces().List(ctx, metav1.ListOptions{ - LabelSelector: LegacyUnlabeledSelector(), - }) - if err != nil { - return fmt.Errorf("list unlabeled managed namespaces: %w", err) - } - var errs []error - for i := range list.Items { - ns := &list.Items[i] - // Defense in depth: a selector over-return (or a client that ignores - // selectors) must not claim unmanaged or already-labeled namespaces. - if !hasManagementLabels(ns) || ns.Labels[InstanceLabel] != "" { - continue - } - if !isGatewayWorkloadName(ns.Name) { - continue - } - if err := EnsureManagedNamespace(ctx, client, ns.Name, instance); err != nil { - errs = append(errs, fmt.Errorf("claim unlabeled namespace %s: %w", ns.Name, err)) - continue - } - log.Printf("INFO claimed unlabeled legacy namespace %s for instance %s", ns.Name, instance) - } - return errors.Join(errs...) -} - // ManagedNamespaceLabels is the label set stamped on namespaces this instance // creates and reconciles. func ManagedNamespaceLabels(instance string) map[string]string { diff --git a/components/control-plane/internal/gateway/namespace_test.go b/components/control-plane/internal/gateway/namespace_test.go index 3a4259a7..29fd956f 100644 --- a/components/control-plane/internal/gateway/namespace_test.go +++ b/components/control-plane/internal/gateway/namespace_test.go @@ -67,14 +67,6 @@ func TestManagedNamespaceSelector(t *testing.T) { } } -func TestLegacyUnlabeledSelector(t *testing.T) { - got := LegacyUnlabeledSelector() - want := "hypershell.redhat.io/managed=true,app.kubernetes.io/managed-by=hypershell-control-plane,!hypershell.redhat.io/instance" - if got != want { - t.Errorf("LegacyUnlabeledSelector() = %q, want %q", got, want) - } -} - func TestIsManagedNamespace(t *testing.T) { tests := []struct { name string @@ -273,85 +265,6 @@ func TestEnsureManagedNamespace(t *testing.T) { }) } -func TestBackfillInstanceLabels(t *testing.T) { - ctx := context.Background() - - t.Run("stamps unlabeled gateway namespaces", func(t *testing.T) { - unlabeled := managedNamespaceForInstance("openshell-legacy", "", nil) - client := fake.NewSimpleClientset(unlabeled) - if err := BackfillInstanceLabels(ctx, client, "hypershell"); err != nil { - t.Fatalf("BackfillInstanceLabels() error = %v", err) - } - got, err := client.CoreV1().Namespaces().Get(ctx, "openshell-legacy", metav1.GetOptions{}) - if err != nil { - t.Fatalf("get namespace: %v", err) - } - if got.Labels[InstanceLabel] != "hypershell" { - t.Errorf("instance label = %q, want hypershell", got.Labels[InstanceLabel]) - } - }) - - t.Run("does not claim unlabeled ManagedDatabase namespaces", func(t *testing.T) { - db := managedNamespaceForInstance("openshell-db-a1b2c3d4e5f67890", "", nil) - client := fake.NewSimpleClientset(db) - if err := BackfillInstanceLabels(ctx, client, "hypershell"); err != nil { - t.Fatalf("BackfillInstanceLabels() error = %v", err) - } - got, err := client.CoreV1().Namespaces().Get(ctx, "openshell-db-a1b2c3d4e5f67890", metav1.GetOptions{}) - if err != nil { - t.Fatalf("get namespace: %v", err) - } - if _, ok := got.Labels[InstanceLabel]; ok { - t.Errorf("ManagedDatabase namespace was claimed, want unlabeled") - } - }) - - t.Run("does not overwrite a foreign instance label", func(t *testing.T) { - foreign := managedNamespaceForInstance("openshell-stage", "hypershell-stage", nil) - client := fake.NewSimpleClientset(foreign) - if err := BackfillInstanceLabels(ctx, client, "hypershell"); err != nil { - t.Fatalf("BackfillInstanceLabels() error = %v", err) - } - got, err := client.CoreV1().Namespaces().Get(ctx, "openshell-stage", metav1.GetOptions{}) - if err != nil { - t.Fatalf("get namespace: %v", err) - } - if got.Labels[InstanceLabel] != "hypershell-stage" { - t.Errorf("instance label = %q, want hypershell-stage", got.Labels[InstanceLabel]) - } - }) - - t.Run("refuses an empty instance identity", func(t *testing.T) { - unlabeled := managedNamespaceForInstance("openshell-legacy", "", nil) - client := fake.NewSimpleClientset(unlabeled) - if err := BackfillInstanceLabels(ctx, client, ""); err == nil { - t.Fatalf("BackfillInstanceLabels() error = nil, want empty instance error") - } - got, err := client.CoreV1().Namespaces().Get(ctx, "openshell-legacy", metav1.GetOptions{}) - if err != nil { - t.Fatalf("get namespace: %v", err) - } - if _, ok := got.Labels[InstanceLabel]; ok { - t.Errorf("empty instance claimed a namespace, want unlabeled") - } - }) - - t.Run("does not claim an unmanaged openshell namespace", func(t *testing.T) { - unmanaged := &corev1.Namespace{ObjectMeta: metav1.ObjectMeta{Name: "openshell-shared"}} - client := fake.NewSimpleClientset(unmanaged) - if err := BackfillInstanceLabels(ctx, client, "hypershell"); err != nil { - t.Fatalf("BackfillInstanceLabels() error = %v", err) - } - got, err := client.CoreV1().Namespaces().Get(ctx, "openshell-shared", metav1.GetOptions{}) - if err != nil { - t.Fatalf("get namespace: %v", err) - } - if len(got.Labels) != 0 { - t.Errorf("unmanaged namespace was labeled %v, want untouched", got.Labels) - } - }) -} - func TestMarkGCEligible(t *testing.T) { ctx := context.Background() now := time.Date(2026, 8, 17, 12, 0, 0, 0, time.UTC) diff --git a/components/control-plane/internal/reconciler/namespace.go b/components/control-plane/internal/reconciler/namespace.go index 66f18962..eeac0eac 100644 --- a/components/control-plane/internal/reconciler/namespace.go +++ b/components/control-plane/internal/reconciler/namespace.go @@ -45,14 +45,13 @@ const ( // this control-plane instance created but that no longer have a live Gateway in // this instance's API server. Other HyperShell instances on the same cluster are // ignored: the sweep selects on hypershell.redhat.io/instance=. -// At the start of each sweep it claims unlabeled legacy gateway namespaces -// (both management labels, no instance label) by stamping this instance, so a -// missed-delete orphan from before the instance label existed is not left -// invisible to GC. This reaps namespaces orphaned by a delete event missed -// while the control plane was down, and namespaces whose gateway failed to -// bootstrap and was then deleted. Reaping is best-effort and idempotent, and is -// delayed by a grace period recorded on the namespace itself so it survives -// restarts. +// Legacy gateway namespaces that predate the instance label must be labeled +// manually (both management labels, no instance label) before they become +// visible to a sweep; this reconciler never claims them itself. This reaps +// namespaces orphaned by a delete event missed while the control plane was +// down, and namespaces whose gateway failed to bootstrap and was then deleted. +// Reaping is best-effort and idempotent, and is delayed by a grace period +// recorded on the namespace itself so it survives restarts. // // See openshell-gateway-namespace-gc.spec.md (HYPERSHELL-78). type NamespaceGCReconciler struct { @@ -136,19 +135,6 @@ func (r *NamespaceGCReconciler) reconcileOnce(ctx context.Context) { return } - // Claim pre-instance-label leftover gateway namespaces so a missed-delete - // orphan (two management labels, no instance label, no live Gateway) is - // visible to the instance-scoped sweep below. Live unlabeled namespaces - // are also stamped here; the live set then keeps them from being orphaned. - // Failure to claim must not abort the labeled sweep: already-labeled - // orphans can still be reaped. Foreign instance labels are never overwritten. - backfillCtx, backfillCancel := context.WithTimeout(ctx, namespaceListTimeout) - if err := gateway.BackfillInstanceLabels(backfillCtx, r.client, r.cpNamespace); err != nil { - tickErr = errors.Join(tickErr, err) - log.Printf("WARN namespace gc: backfill unlabeled instance labels: %v", err) - } - backfillCancel() - selector, err := gateway.ManagedNamespaceSelector(r.cpNamespace) if err != nil { tickErr = err diff --git a/components/control-plane/internal/reconciler/namespace_test.go b/components/control-plane/internal/reconciler/namespace_test.go index 9ef9d3d6..26c05ed6 100644 --- a/components/control-plane/internal/reconciler/namespace_test.go +++ b/components/control-plane/internal/reconciler/namespace_test.go @@ -308,11 +308,7 @@ func TestReconcileOnce_OnlySweepsThisInstance(t *testing.T) { gateway.ManagedLabel: gateway.ManagedLabelValue, gateway.InstanceLabel: "hypershell-stage", }, nil) - unlabeled := nsWithLabels("openshell-legacy", map[string]string{ - gateway.ManagedByLabel: gateway.ManagedByValue, - gateway.ManagedLabel: gateway.ManagedLabelValue, - }, nil) - client := fake.NewSimpleClientset(own, foreign, unlabeled) + client := fake.NewSimpleClientset(own, foreign) r := newTestGC(client, now) r.reconcileOnce(ctx) @@ -325,17 +321,6 @@ func TestReconcileOnce_OnlySweepsThisInstance(t *testing.T) { t.Errorf("this instance's orphan was not stamped gc-eligible-since") } - legacy, err := client.CoreV1().Namespaces().Get(ctx, "openshell-legacy", metav1.GetOptions{}) - if err != nil { - t.Fatalf("get unlabeled legacy namespace: %v", err) - } - if legacy.Labels[gateway.InstanceLabel] != "hypershell" { - t.Errorf("unlabeled legacy namespace instance = %q, want hypershell after backfill", legacy.Labels[gateway.InstanceLabel]) - } - if legacy.Annotations[gateway.GCEligibleSinceAnnotation] != now.Format(time.RFC3339) { - t.Errorf("unlabeled legacy orphan was not stamped gc-eligible-since after backfill") - } - foreignUpdated, err := client.CoreV1().Namespaces().Get(ctx, "openshell-stage", metav1.GetOptions{}) if err != nil { t.Fatalf("get foreign namespace: %v", err) @@ -348,73 +333,33 @@ func TestReconcileOnce_OnlySweepsThisInstance(t *testing.T) { } } -func TestReconcileOnce_BackfillDoesNotOrphanLiveUnlabeled(t *testing.T) { +// Legacy gateway namespaces that predate the instance label are invisible to +// the sweep until labeled manually: the reconciler must never touch them +// itself, regardless of how long they have existed or whether they carry a +// stale gc-eligible-since annotation from before labeling. +func TestReconcileOnce_UnlabeledLegacyNamespaceIsIgnored(t *testing.T) { ctx := context.Background() now := time.Date(2026, 8, 17, 12, 0, 0, 0, time.UTC) unlabeled := nsWithLabels("openshell-legacy", map[string]string{ gateway.ManagedByLabel: gateway.ManagedByValue, gateway.ManagedLabel: gateway.ManagedLabelValue, - }, nil) + }, map[string]string{ + gateway.GCEligibleSinceAnnotation: now.Add(-20 * time.Minute).Format(time.RFC3339), + }) client := fake.NewSimpleClientset(unlabeled) r := newTestGC(client, now) - r.liveNamespaces = func(context.Context) (map[string]struct{}, error) { - return map[string]struct{}{"openshell-legacy": {}}, nil - } r.reconcileOnce(ctx) - got, err := client.CoreV1().Namespaces().Get(ctx, "openshell-legacy", metav1.GetOptions{}) - if err != nil { - t.Fatalf("get namespace: %v", err) - } - if got.Labels[gateway.InstanceLabel] != "hypershell" { - t.Errorf("live unlabeled namespace instance = %q, want hypershell after backfill", got.Labels[gateway.InstanceLabel]) + if !nsExists(t, client, "openshell-legacy") { + t.Fatalf("unlabeled legacy namespace reaped, want retained until manually labeled") } - if _, ok := got.Annotations[gateway.GCEligibleSinceAnnotation]; ok { - t.Errorf("live unlabeled namespace was marked orphaned, want retained") - } -} - -func TestReconcileOnce_BackfillDoesNotClaimUnlabeledDatabase(t *testing.T) { - ctx := context.Background() - now := time.Date(2026, 8, 17, 12, 0, 0, 0, time.UTC) - db := nsWithLabels("openshell-db-a1b2c3d4e5f67890", map[string]string{ - gateway.ManagedByLabel: gateway.ManagedByValue, - gateway.ManagedLabel: gateway.ManagedLabelValue, - }, nil) - client := fake.NewSimpleClientset(db) - r := newTestGC(client, now) - - r.reconcileOnce(ctx) - - got, err := client.CoreV1().Namespaces().Get(ctx, "openshell-db-a1b2c3d4e5f67890", metav1.GetOptions{}) + got, err := client.CoreV1().Namespaces().Get(ctx, "openshell-legacy", metav1.GetOptions{}) if err != nil { t.Fatalf("get namespace: %v", err) } if _, ok := got.Labels[gateway.InstanceLabel]; ok { - t.Errorf("unlabeled ManagedDatabase namespace was claimed, want unlabeled") - } - if _, ok := got.Annotations[gateway.GCEligibleSinceAnnotation]; ok { - t.Errorf("ManagedDatabase namespace was treated as an orphan, want ignored") - } -} - -func TestReconcileOnce_UnlabeledOrphanPastGraceIsReaped(t *testing.T) { - ctx := context.Background() - now := time.Date(2026, 8, 17, 12, 0, 0, 0, time.UTC) - unlabeled := nsWithLabels("openshell-legacy", map[string]string{ - gateway.ManagedByLabel: gateway.ManagedByValue, - gateway.ManagedLabel: gateway.ManagedLabelValue, - }, map[string]string{ - gateway.GCEligibleSinceAnnotation: now.Add(-20 * time.Minute).Format(time.RFC3339), - }) - client := fake.NewSimpleClientset(unlabeled) - r := newTestGC(client, now) - - r.reconcileOnce(ctx) - - if nsExists(t, client, "openshell-legacy") { - t.Fatalf("unlabeled legacy orphan past grace retained, want reaped after backfill") + t.Errorf("unlabeled legacy namespace was claimed, want unlabeled") } } diff --git a/deploy/base/api-server.yaml b/deploy/base/api-server.yaml index d503669a..4676855f 100644 --- a/deploy/base/api-server.yaml +++ b/deploy/base/api-server.yaml @@ -91,10 +91,17 @@ spec: # the sibling control-plane -> api-server watch channel (no mTLS). # Use a cluster-local FQDN. grpc-go's DNS resolver does not apply # kube-DNS search domains, so the short name hypershell-controller - # fails even when libc lookup (nc) succeeds. OpenShift rewrite maps - # hypershell-system to OPENSHIFT_NAMESPACE. + # fails even when libc lookup (nc) succeeds. POD_NAMESPACE is this + # pod's namespace (downward API), the same source as the + # controller's HYPERSHELL_NAMESPACE instance identity. Kubelet + # expands $(POD_NAMESPACE) so the dial target tracks the Deployment + # namespace without a hardcoded hypershell-system segment. + - name: POD_NAMESPACE + valueFrom: + fieldRef: + fieldPath: metadata.namespace - name: HYPERSHELL_SERVICE_ACCOUNT_PROVISIONER_ADDR - value: "hypershell-controller.hypershell-system.svc.cluster.local:9443" + value: "hypershell-controller.$(POD_NAMESPACE).svc.cluster.local:9443" - name: DB_SSLMODE value: require startupProbe: diff --git a/deploy/openshift/kustomization.yaml b/deploy/openshift/kustomization.yaml index 57193169..d10730b3 100644 --- a/deploy/openshift/kustomization.yaml +++ b/deploy/openshift/kustomization.yaml @@ -95,9 +95,15 @@ patches: - name: API_ENV value: "development_oidc" # Re-declare the base env: this JSON6902 add REPLACES the whole env - # array, so the base values would otherwise be dropped. + # array, so the base values would otherwise be dropped. POD_NAMESPACE + # must precede the provisioner address so kubelet can expand it. + # Same downward-API source as the controller HYPERSHELL_NAMESPACE. + - name: POD_NAMESPACE + valueFrom: + fieldRef: + fieldPath: metadata.namespace - name: HYPERSHELL_SERVICE_ACCOUNT_PROVISIONER_ADDR - value: "hypershell-controller.hypershell-system.svc.cluster.local:9443" + value: "hypershell-controller.$(POD_NAMESPACE).svc.cluster.local:9443" - name: DB_SSLMODE value: require - name: JWK_CERT_URL diff --git a/scripts/cluster/drivers/openshift.sh b/scripts/cluster/drivers/openshift.sh index 4afe586a..18721816 100755 --- a/scripts/cluster/drivers/openshift.sh +++ b/scripts/cluster/drivers/openshift.sh @@ -1260,31 +1260,38 @@ cluster_status() { fi } -login_internal_registry() { - local registry - if [[ -n "${OPENSHIFT_IMAGE_REGISTRY:-}" ]]; then - registry="${OPENSHIFT_IMAGE_REGISTRY}" - else - registry="$(oc_cli registry info 2>/dev/null || true)" - fi - if [[ -z "${registry}" ]]; then - error "Could not discover the OpenShift internal registry." - error "Set OPENSHIFT_IMAGE_REGISTRY to a registry the cluster can pull from, then retry." - exit 1 - fi - OPENSHIFT_PUSH_REGISTRY="${registry}" - OPENSHIFT_PULL_REGISTRY="$(oc_cli registry info --internal 2>/dev/null || printf '%s' "${registry}")" - info "Logging in to registry ${OPENSHIFT_PUSH_REGISTRY}..." - if oc_cli registry login --registry "${OPENSHIFT_PUSH_REGISTRY}" >/dev/null 2>&1; then +swap_registry_prefix() { + require_swap_registry || exit 1 + local prefix="${SWAP_REGISTRY#https://}" + prefix="${prefix#http://}" + printf '%s' "${prefix%/}" +} + +swap_registry_host() { + local prefix + prefix="$(swap_registry_prefix)" + printf '%s' "${prefix%%/*}" +} + +login_swap_registry() { + local prefix host + prefix="$(swap_registry_prefix)" + OPENSHIFT_PUSH_REGISTRY="${prefix}" + OPENSHIFT_PULL_REGISTRY="${prefix}" + host="$(swap_registry_host)" + info "SWAP_REGISTRY=${prefix} (host ${host})" + if [[ -n "${SWAP_REPOSITORY:-}" ]]; then + info "SWAP_REPOSITORY=${SWAP_REPOSITORY}" + fi + if [[ -n "$(resolved_pull_secret_path)" ]]; then + login_registry_with_pull_secret "${host}" || exit 1 return 0 fi - local user token - user="$(oc_cli whoami)" - token="$(oc_cli whoami -t)" - if ${CONTAINER_ENGINE} login -u "${user}" -p "${token}" "${OPENSHIFT_PUSH_REGISTRY}" >/dev/null; then + if ${CONTAINER_ENGINE} login --get-login "${host}" >/dev/null 2>&1; then return 0 fi - error "Failed to log in to ${OPENSHIFT_PUSH_REGISTRY}. Set OPENSHIFT_IMAGE_REGISTRY or fix oc registry login." + error "Not logged in to ${host}." + error "Set PULL_SECRET to a kubernetes.io/dockerconfigjson Secret YAML (KIND_PULL_SECRET is still accepted), or run: ${CONTAINER_ENGINE} login ${host}" exit 1 } @@ -1295,29 +1302,49 @@ push_component_image() { commit="$(git -C "${REPO_ROOT}" rev-parse --short HEAD 2>/dev/null || echo unknown)" local tag="${commit}-${OPENSHIFT_NAMESPACE}" tag="$(printf '%s' "${tag}" | tr '[:upper:]' '[:lower:]' | sed -E 's/[^a-z0-9._-]+/-/g')" - local imagestream="hypershell-${component}" - oc_cli create imagestream "${imagestream}" -n "${OPENSHIFT_NAMESPACE}" --dry-run=client -o yaml \ - | oc_cli apply -f - >/dev/null + local repo + repo="$(swap_image_repository "${component}")" || exit 1 + local push_ref="${repo}:${tag}" + local target_arch + target_arch="$(swap_target_goarch)" || exit 1 - local push_ref="${OPENSHIFT_PUSH_REGISTRY}/${OPENSHIFT_NAMESPACE}/${imagestream}:${tag}" - info "Building ${component} from working tree..." + info "Building ${component} from working tree for linux/${target_arch}..." + if [[ "${component}" == "web-console" ]]; then + warn "web-console cannot Go-cross-compile; this Node image matches the laptop unless you rebuild on ${target_arch}." + fi ${CONTAINER_ENGINE} build -t "${LOCAL_IMAGE}" \ -f "${REPO_ROOT}/${DOCKERFILE}" ${BUILD_ARGS[@]+"${BUILD_ARGS[@]}"} \ + --build-arg "TARGETARCH=${target_arch}" \ + --build-arg "TARGETOS=linux" \ "${REPO_ROOT}/${BUILD_CONTEXT}" ${CONTAINER_ENGINE} tag "${LOCAL_IMAGE}" "${push_ref}" info "Pushing ${push_ref}..." - ${CONTAINER_ENGINE} push "${push_ref}" - - local digest="" - digest="$(oc_cli get imagestreamtag "${imagestream}:${tag}" -n "${OPENSHIFT_NAMESPACE}" \ - -o jsonpath='{.image.metadata.name}' 2>/dev/null || true)" - local pull_ref - if [[ -n "${digest}" ]]; then - pull_ref="${OPENSHIFT_PULL_REGISTRY}/${OPENSHIFT_NAMESPACE}/${imagestream}@${digest}" + # Pin the registry manifest digest from this push, not `inspect` of the + # local image. Local digests (and the blob/config SHAs in podman progress) + # are not the digest Quay stores, so a cluster pull by that digest fails + # with manifest unknown. `podman push --digestfile` writes the digest the + # registry accepted. Docker prints `digest: sha256:...` on stdout instead. + local digestfile push_log digest="" + digestfile="$(mktemp)" + push_log="$(mktemp)" + if ${CONTAINER_ENGINE} push --help 2>&1 | grep -q -- '--digestfile'; then + ${CONTAINER_ENGINE} push --digestfile="${digestfile}" "${push_ref}" + digest="$(tr -d '[:space:]' < "${digestfile}")" + else + ${CONTAINER_ENGINE} push "${push_ref}" >"${push_log}" 2>&1 || { + cat "${push_log}" + rm -f "${digestfile}" "${push_log}" + exit 1 + } + cat "${push_log}" + digest="$(registry_digest_from_push_log < "${push_log}")" + fi + rm -f "${digestfile}" "${push_log}" + if [[ "${digest}" == sha256:* ]]; then + OPENSHIFT_SWAPPED_IMAGE="${repo}@${digest}" else - pull_ref="${OPENSHIFT_PULL_REGISTRY}/${OPENSHIFT_NAMESPACE}/${imagestream}:${tag}" + OPENSHIFT_SWAPPED_IMAGE="${push_ref}" fi - OPENSHIFT_SWAPPED_IMAGE="${pull_ref}" } rollout_component_image() { @@ -1350,7 +1377,7 @@ component_swap() { exit 1 fi refuse_foreign_namespace "${OPENSHIFT_NAMESPACE}" >/dev/null - login_internal_registry + login_swap_registry push_component_image "${component}" info "Rolling out ${component} to ${OPENSHIFT_SWAPPED_IMAGE}" rollout_component_image "${component}" "${OPENSHIFT_SWAPPED_IMAGE}" diff --git a/scripts/cluster/lib.sh b/scripts/cluster/lib.sh index 0605e6f4..c702cea4 100755 --- a/scripts/cluster/lib.sh +++ b/scripts/cluster/lib.sh @@ -97,6 +97,212 @@ is_reserved_cluster_namespace() { return 1 } +# True when a registry hostname only resolves inside the cluster. Laptop +# podman/docker cannot push to these names. +registry_host_is_cluster_local() { + local host="${1:-}" + host="${host#https://}" + host="${host#http://}" + host="${host%%/*}" + case "${host}" in + *.svc|*.svc:*|*.cluster.local|*.cluster.local:*) return 0 ;; + esac + return 1 +} + +# SWAP_REGISTRY is the laptop-push org prefix for OpenShift component swaps +# (for example quay.io/). IMAGE_REGISTRY is the baseline/pull path and is +# never used as the swap destination. Repo names default to hypershell-api-server, +# hypershell-controller, and hypershell-web-console; SWAP_REPOSITORY overrides +# the repo name for the current swap. +require_swap_registry() { + if [[ -z "${SWAP_REGISTRY:-}" ]]; then + error "SWAP_REGISTRY is unset. Set it to a registry org prefix this cluster can pull (for example quay.io/)." + error "IMAGE_REGISTRY is only for baseline images; swaps do not push there." + return 1 + fi + local prefix="${SWAP_REGISTRY#https://}" + prefix="${prefix#http://}" + prefix="${prefix%/}" + if registry_host_is_cluster_local "${prefix}"; then + error "SWAP_REGISTRY=${SWAP_REGISTRY} is only reachable inside the cluster." + error "Set SWAP_REGISTRY to a Quay org prefix this cluster can pull (for example quay.io/)." + return 1 + fi + if [[ "${prefix}" != */* ]]; then + error "SWAP_REGISTRY must be a registry org prefix (for example quay.io/), not '${SWAP_REGISTRY}'." + return 1 + fi +} + +# GOARCH for OpenShift swap images. Laptop architecture is not used: ROSA and +# most OpenShift nodes are amd64, while developer laptops may be arm64. +# SWAP_PLATFORM (linux/amd64 or linux/arm64) wins; otherwise the first node's +# architecture is used. +swap_target_goarch() { + local raw="${SWAP_PLATFORM:-${SWAP_ARCH:-}}" + if [[ -z "${raw}" ]]; then + raw="$(oc get nodes -o jsonpath='{.items[0].status.nodeInfo.architecture}' 2>/dev/null || true)" + fi + case "${raw}" in + linux/amd64|amd64|x86_64) printf 'amd64' ;; + linux/arm64|arm64|aarch64) printf 'arm64' ;; + '') + error "Could not detect OpenShift node architecture. Set SWAP_PLATFORM=linux/amd64 (or linux/arm64) to match the cluster nodes." + return 1 + ;; + *) + error "Unsupported swap architecture '${raw}'. Set SWAP_PLATFORM=linux/amd64 or linux/arm64." + return 1 + ;; + esac +} + +swap_default_repository() { + case "${1:-}" in + api-server) printf 'hypershell-api-server' ;; + control-plane) printf 'hypershell-controller' ;; + web-console) printf 'hypershell-web-console' ;; + *) + error "Unknown component: ${1:-}" + return 1 + ;; + esac +} + +swap_repository_for_component() { + if [[ -n "${SWAP_REPOSITORY:-}" ]]; then + printf '%s' "${SWAP_REPOSITORY}" + return 0 + fi + swap_default_repository "$1" +} + +swap_image_repository() { + local prefix repo_name + require_swap_registry || return 1 + prefix="${SWAP_REGISTRY#https://}" + prefix="${prefix#http://}" + prefix="${prefix%/}" + repo_name="$(swap_repository_for_component "$1")" || return 1 + printf '%s/%s' "${prefix}" "${repo_name}" +} + +# Registry manifest digest from a docker-style push log (`digest: sha256:...`). +# Ignore unanchored sha256 values: podman progress prints blob and config +# digests that are not the digest the registry stores for the tag. +registry_digest_from_push_log() { + sed -nE 's/.*digest: (sha256:[0-9a-f]{64}).*/\1/p' | tail -1 +} + +# PULL_SECRET is the canonical path to a kubernetes.io/dockerconfigjson Secret +# YAML (or a raw Docker config JSON). KIND_PULL_SECRET remains an alias. +resolved_pull_secret_path() { + printf '%s' "${PULL_SECRET:-${KIND_PULL_SECRET:-}}" +} + +# Print {"username":"...","password":"..."} for registry host from a pull secret. +registry_auth_json_for_host() { + local file="${1:-}" host="${2:-}" + if [[ -z "${file}" || ! -f "${file}" ]]; then + error "Pull secret file is missing: ${file:-'(empty path)'}" + return 1 + fi + python3 - "${file}" "${host}" <<'PY' +import base64, json, re, sys + +path, host = sys.argv[1], sys.argv[2] +text = open(path, encoding="utf-8").read() + +def load_dockerconfig(raw: str): + raw = raw.strip() + try: + data = json.loads(raw) + except json.JSONDecodeError: + data = None + if isinstance(data, dict) and "auths" in data: + return data + if isinstance(data, dict): + b64 = (data.get("data") or {}).get(".dockerconfigjson") + if b64: + return json.loads(base64.b64decode(b64)) + match = re.search( + r"^\s*\.dockerconfigjson:\s*[\"']?([A-Za-z0-9+/=]+)[\"']?\s*$", + raw, + re.M, + ) + if not match: + raise SystemExit("no dockerconfigjson in pull secret") + blob = match.group(1) + decoded = base64.b64decode(blob) + try: + return json.loads(decoded.decode("utf-8")) + except (UnicodeDecodeError, json.JSONDecodeError) as exc: + raise SystemExit( + "pull secret .dockerconfigjson is not JSON; the YAML value may be truncated" + ) from exc + +def credentials(entry: dict): + user, password = entry.get("username"), entry.get("password") + if user and password: + return user, password + auth = entry.get("auth") + if not auth: + return None + decoded = base64.b64decode(auth).decode("utf-8") + user, password = decoded.split(":", 1) + return user, password + +cfg = load_dockerconfig(text) +auths = cfg.get("auths") or {} +candidates = [ + host, + f"https://{host}", + f"http://{host}", + f"https://{host}/v1/", + f"https://{host}/v2/", +] +entry = None +for key in candidates: + if key in auths: + entry = auths[key] + break +if entry is None: + for key, value in auths.items(): + if host in key: + entry = value + break +if entry is None: + raise SystemExit(f"pull secret has no auth for {host}") +pair = credentials(entry) +if not pair: + raise SystemExit(f"pull secret auth for {host} has no username/password") +json.dump({"username": pair[0], "password": pair[1]}, sys.stdout) +PY +} + +login_registry_with_pull_secret() { + local host="$1" + local file + file="$(resolved_pull_secret_path)" + if [[ -z "${file}" ]]; then + error "PULL_SECRET is unset. Set it to a kubernetes.io/dockerconfigjson Secret YAML (KIND_PULL_SECRET is still accepted)." + return 1 + fi + local auth_json user password + if ! auth_json="$(registry_auth_json_for_host "${file}" "${host}")"; then + error "Could not read ${host} credentials from ${file}." + return 1 + fi + user="$(printf '%s' "${auth_json}" | python3 -c 'import json,sys; print(json.load(sys.stdin)["username"])')" + password="$(printf '%s' "${auth_json}" | python3 -c 'import json,sys; print(json.load(sys.stdin)["password"])')" + info "Logging in to ${host} with PULL_SECRET" + if ! printf '%s' "${password}" | ${CONTAINER_ENGINE} login --username "${user}" --password-stdin "${host}" >/dev/null; then + error "Failed to log in to ${host} with credentials from ${file}." + return 1 + fi +} + # Strip a Gateway listener hostname (e.g. *.openshell.example.com) down to the # tenant base domain the control plane expects (openshell.example.com). gateway_base_domain_from_hostname() { diff --git a/scripts/cluster/lib_test.sh b/scripts/cluster/lib_test.sh index f058d606..b512c587 100755 --- a/scripts/cluster/lib_test.sh +++ b/scripts/cluster/lib_test.sh @@ -51,6 +51,83 @@ assert_fail "uppercase rejected" validate_rfc1123_label "Alice" 54 assert_fail "underscore rejected" validate_rfc1123_label "alice_dev" 54 assert_eq "tok" "$(printf '%s' '{"access_token":"tok","expires_in":60}' | json_string_field access_token)" "json_string_field access_token" assert_eq "abc-id" "$(printf '%s' '[{"id":"abc-id","clientId":"hypershell-frontend"}]' | json_first_id)" "json_first_id" +assert_ok "internal registry svc:port is cluster-local" \ + registry_host_is_cluster_local 'image-registry.openshift-image-registry.svc:5000' +assert_ok "cluster.local registry is cluster-local" \ + registry_host_is_cluster_local 'image-registry.openshift-image-registry.svc.cluster.local:5000' +assert_fail "quay.io is not cluster-local" registry_host_is_cluster_local 'quay.io' +assert_fail "apps route is not cluster-local" \ + registry_host_is_cluster_local 'default-route-openshift-image-registry.apps.example.com' +( + unset SWAP_REGISTRY IMAGE_REGISTRY + assert_fail "unset SWAP_REGISTRY is an error" require_swap_registry +) +SWAP_REGISTRY='quay.io/example' \ + assert_ok "set SWAP_REGISTRY org prefix is accepted" require_swap_registry +SWAP_REGISTRY='quay.io' \ + assert_fail "SWAP_REGISTRY without org is refused" require_swap_registry +SWAP_REGISTRY='image-registry.openshift-image-registry.svc:5000' \ + assert_fail "cluster-local SWAP_REGISTRY is refused" require_swap_registry +assert_eq "hypershell-api-server" "$(swap_default_repository api-server)" "default api-server repo" +assert_eq "hypershell-controller" "$(swap_default_repository control-plane)" "default control-plane repo" +assert_eq "hypershell-web-console" "$(swap_default_repository web-console)" "default web-console repo" +assert_eq "hypershell-api-server" "$(unset SWAP_REPOSITORY; swap_repository_for_component api-server)" \ + "api-server repo without override" +assert_eq "custom-api" "$(SWAP_REPOSITORY=custom-api swap_repository_for_component api-server)" \ + "SWAP_REPOSITORY overrides repo name" +assert_eq "quay.io/alice/hypershell-api-server" \ + "$(SWAP_REGISTRY=quay.io/alice SWAP_REPOSITORY= swap_image_repository api-server)" \ + "swap image is org prefix plus default repo" +assert_eq "quay.io/alice/custom-api" \ + "$(SWAP_REGISTRY=quay.io/alice SWAP_REPOSITORY=custom-api swap_image_repository api-server)" \ + "swap image uses SWAP_REPOSITORY override" +assert_eq "amd64" "$(SWAP_PLATFORM=linux/amd64 swap_target_goarch)" "SWAP_PLATFORM linux/amd64" +assert_eq "amd64" "$(SWAP_ARCH=x86_64 SWAP_PLATFORM= swap_target_goarch)" "SWAP_ARCH x86_64" +assert_eq "arm64" "$(SWAP_PLATFORM=linux/arm64 swap_target_goarch)" "SWAP_PLATFORM linux/arm64" +SWAP_PLATFORM=linux/ppc64le assert_fail "unsupported SWAP_PLATFORM" swap_target_goarch +unset SWAP_PLATFORM SWAP_ARCH +assert_eq "sha256:d3f6ac0a7627fee89b55f34745e09fc64d0073e807719a66f6b4534a96541eb6" \ + "$(printf '%s\n' \ + 'Copying blob sha256:5bef08742407efd622d243692b79ba0055383bbce12900324f75e56f589aedb0' \ + 'Copying config sha256:ad4686094d8f0186ec8249fc4917b71faa2c1030d7b5a025c29f26e19d95c156' \ + 'Writing manifest to image destination' \ + '6014ae9-hypershell-e2e-test: digest: sha256:d3f6ac0a7627fee89b55f34745e09fc64d0073e807719a66f6b4534a96541eb6 size: 1927' \ + | registry_digest_from_push_log)" \ + "push log parser takes digest: line not blob/config SHAs" +assert_eq "" \ + "$(printf '%s\n' \ + 'Copying blob sha256:5bef08742407efd622d243692b79ba0055383bbce12900324f75e56f589aedb0' \ + 'Copying config sha256:ad4686094d8f0186ec8249fc4917b71faa2c1030d7b5a025c29f26e19d95c156' \ + 'Writing manifest to image destination' \ + 'Storing signatures' \ + | registry_digest_from_push_log)" \ + "podman progress without a digest: line is not a registry digest" +assert_eq "/from-pull" \ + "$(PULL_SECRET=/from-pull KIND_PULL_SECRET=/from-kind resolved_pull_secret_path)" \ + "PULL_SECRET wins over KIND_PULL_SECRET" +assert_eq "/from-kind" \ + "$(PULL_SECRET= KIND_PULL_SECRET=/from-kind resolved_pull_secret_path)" \ + "KIND_PULL_SECRET is the PULL_SECRET alias" +_pull_auth="$(printf '%s' 'alice:s3cret' | base64 | tr -d '\n')" +_pull_dc="$(printf '%s' "{\"auths\":{\"quay.io\":{\"auth\":\"${_pull_auth}\"}}}" | base64 | tr -d '\n')" +_pull_secret="$(mktemp)" +cat > "${_pull_secret}" < **Database images:** Two environment variables control PostgreSQL images for the two CNPG provisioning paths: > - `HYPERSHELL_DATABASE_IMAGE` - configures the API server's static `hypershell-db` CNPG Cluster. `make kind-up` patches the Cluster with this image after applying manifests. When unset, CNPG uses its built-in default image. @@ -786,7 +794,8 @@ The system SHALL deploy a Jaeger all-in-one instance in the local environment an | `KIND_HOT_RELOAD` | `true` | Hot reload for supported components; set to `false` to disable | | `KIND_HOST_MOUNT_PATH` | Repository root (`git rev-parse --show-toplevel`) | Host directory mounted into Kind nodes for hot reload | | `KIND_KEYCLOAK_URL` | (unset - deploy local) | External Keycloak issuer URL; skips local deployment when set | -| `KIND_PULL_SECRET` | (unset) | Path to a Kubernetes pull secret YAML file; applied to the target namespace for HI image access | +| `PULL_SECRET` | (unset) | Path to a Kubernetes pull secret YAML file (`kubernetes.io/dockerconfigjson`); applied to the Kind target namespace and used to log the container engine in for OpenShift swaps | +| `KIND_PULL_SECRET` | (unset) | Alias for `PULL_SECRET` | | `IMAGE_REGISTRY` | `quay.io/redhat-services-prod/hcm-eng-prod-tenant/hypershell-main` | Container registry path for baseline images | | `IMAGE_TAG` | `latest` | Image tag for baseline images | | `LOCAL_IMAGES` | (unset - pull from registry) | Set to `true` to build images locally instead of pulling from registry | diff --git a/specs/platform/openshell-gateway-namespace-gc.spec.md b/specs/platform/openshell-gateway-namespace-gc.spec.md index ab02ab0d..54d2ca93 100644 --- a/specs/platform/openshell-gateway-namespace-gc.spec.md +++ b/specs/platform/openshell-gateway-namespace-gc.spec.md @@ -280,15 +280,6 @@ controllers on the same cluster SHALL NOT share an instance identity. - THEN it SHALL NOT overwrite the instance label - AND it SHALL NOT deploy into or delete that namespace -#### Scenario: Periodic GC backfills unlabeled legacy gateway namespaces - -- GIVEN a gateway-prefixed namespace that carries the two management labels - but no instance label -- WHEN the garbage-collection reconciler sweeps -- THEN it SHALL stamp `hypershell.redhat.io/instance` for this instance -- AND it SHALL NOT change an instance label that already identifies a different - instance - ### Requirement: Grace Period Prevents Premature Deletion The control plane SHALL NOT delete an orphaned namespace immediately. It SHALL @@ -424,9 +415,9 @@ real-time guarantee. | GC triggers on orphaning (no live Gateway), not on `phase` being `Degraded`/`Failed` | A gateway that still exists - even if unhealthy - is the health reconciler's and the operator's concern; only the absence of a backing Gateway unambiguously means the namespace is garbage. This avoids reaping a namespace an operator is still debugging. | | Exclude `openshell-db-*` managed namespaces from periodic GC | ManagedDatabase CNPG namespaces share the management labels but are owned by the ManagedDatabase reconciler; the stable `openshell-db-` prefix distinguishes them without requiring a label migration on existing gateway namespaces. | | Require this instance's identity label before periodic GC | Two HyperShell controllers on one cluster share the generic management labels. Without a value unique to that controller (`hypershell.redhat.io/instance=`), instance B's sweep treats instance A's live gateways as orphans (they are absent from B's API server) and would delete them after the grace period. The identity is the controller pod's namespace from the downward API so it cannot be a copied static string. | -| Claim unlabeled legacy gateway namespaces at the start of each GC sweep | The instance label was added after namespaces already existed. `EnsureManagedNamespace` only stamps live Gateways; a missed-delete orphan never hits that path and would leak forever if unlabeled namespaces stayed excluded from the sweep. Each sweep lists namespaces that carry both management labels and no instance label, stamps this instance on gateway-prefixed names, then proceeds with the instance-scoped orphan sweep. Namespaces already labeled for another instance are never claimed. Two HyperShell instances that both still have unlabeled namespaces on the same cluster (neither has been upgraded to stamp the instance label) are unsupported: upgrade both, or label namespaces before enabling GC. | +| Periodic GC never claims unlabeled legacy gateway namespaces | An earlier design had each sweep stamp the instance label onto unlabeled leftovers before evaluating them for orphaning, so a missed-delete orphan predating the instance label would not leak forever. In practice this reclaimed the label on every sweep for a namespace whose instance label never durably stuck (e.g. a naming or list-vs-read inconsistency), which reset `gc-eligible-since` to "now" on every tick and made the namespace look freshly orphaned forever, so it was never reaped. Claiming unlabeled leftovers is now an explicit operator action instead. | | Require BOTH management labels plus this instance's identity before deleting | Defense in depth: even if a label selector over-returns, a namespace not created by this control-plane instance (another HyperShell, a shared namespace, or a pre-existing namespace) is never deleted by periodic GC. | -| Stamp the instance label on create, on live reconcile, and on GC backfill | Periodic GC can only reap namespaces this instance labeled. Stamping at create covers new gateways; stamping on live reconcile migrates this instance's still-live pre-label namespaces; GC backfill migrates unlabeled leftovers that have no live Gateway so the missed-delete path can proceed. | +| Stamp the instance label on create and on live reconcile only | Periodic GC can only reap namespaces this instance labeled. Stamping at create covers new gateways; stamping on live reconcile migrates this instance's still-live pre-label namespaces. Unlabeled leftovers with no live Gateway stay unlabeled and excluded from GC until an operator labels them. | | Grace period persisted on the namespace annotation | The delay must survive control-plane restarts; storing `gc-eligible-since` on the namespace makes the timer durable without a separate store. | | Abort the whole sweep if Gateways cannot be listed | An empty or failed Gateway list would make every managed namespace look orphaned; aborting is the only safe response to avoid mass reaping of live namespaces. | | Delete is best-effort and not gated on sandbox count | Deletion is idempotent - process the delete, remove the namespace, and if it is already gone consider the delete done. The sandbox count is a warning surfaced to the operator, not a backend precondition. | diff --git a/specs/platform/openshift-development.spec.md b/specs/platform/openshift-development.spec.md index 31f53d3d..a13a57fc 100644 --- a/specs/platform/openshift-development.spec.md +++ b/specs/platform/openshift-development.spec.md @@ -444,11 +444,15 @@ kustomize-build time. The overlay SHALL re-declare any base container env the JSON6902 env-array replace would otherwise drop, including `HYPERSHELL_SERVICE_ACCOUNT_PROVISIONER_ADDR`. That address SHALL be the cluster-local FQDN -`hypershell-controller.${OPENSHIFT_NAMESPACE}.svc.cluster.local:9443` -(the overlay stores `hypershell-system`; namespace rewrite substitutes the -assigned project). grpc-go does not apply kube-DNS search domains, so the -short name `hypershell-controller:9443` fails in-cluster even when `nc` to -that short name succeeds. +`hypershell-controller.$(POD_NAMESPACE).svc.cluster.local:9443`, where +`POD_NAMESPACE` is the API server pod's namespace from the downward API +(`metadata.namespace`). The API server and controller share that namespace, so +the dial target matches the controller's `HYPERSHELL_NAMESPACE` instance +identity without a hardcoded `hypershell-system` segment. Namespace rewrite +does not substitute this value; kubelet expands `$(POD_NAMESPACE)` at pod +start. grpc-go does not apply kube-DNS search domains, so the short name +`hypershell-controller:9443` fails in-cluster even when `nc` to that short +name succeeds. This spec defines only where Keycloak lands. The broader isolation of other non-request-serving components (for example the database and observability) into @@ -470,8 +474,9 @@ their own namespaces is out of scope here and belongs to a separate spec. - GIVEN `deploy/openshift/kustomization.yaml` is built - WHEN the rendered API server Deployment is inspected - THEN it SHALL set `API_ENV=development_oidc` -- AND it SHALL retain `HYPERSHELL_SERVICE_ACCOUNT_PROVISIONER_ADDR` as the - cluster-local FQDN for this project's controller Service +- AND it SHALL retain `HYPERSHELL_SERVICE_ACCOUNT_PROVISIONER_ADDR` as + `hypershell-controller.$(POD_NAMESPACE).svc.cluster.local:9443` with + `POD_NAMESPACE` from the downward API (`metadata.namespace`) - AND `make openshift-up` SHALL NOT set `API_ENV` with `oc set env` #### Scenario: Platform workloads can reach Keycloak across the namespace group @@ -507,10 +512,43 @@ component to the baseline registry image. The swap SHALL build the component image from the working tree and make the image available to the cluster. Because OpenShift pulls images from a registry rather -than from a local archive, the OpenShift driver SHALL push the built image to a -registry that the cluster can pull, rather than use the Kind-specific -`kind load image-archive`. The driver SHOULD use the OpenShift internal registry -when it is available, so that the swap does not require an external registry. +than from a local archive, the OpenShift driver SHALL push the built image to +`SWAP_REGISTRY`, a laptop-reachable registry prefix the cluster can pull, rather +than use the Kind-specific `kind load image-archive`. `SWAP_REGISTRY` SHALL be the registry org prefix only (for example +`quay.io/`), not an image repository. The driver SHALL append a default +repository name for the component being swapped: `hypershell-api-server` for +the API server, `hypershell-controller` for the control plane, and +`hypershell-web-console` for the web console. When `SWAP_REPOSITORY` is set, +the driver SHALL use that repository name instead of the component default. +`SWAP_REGISTRY` SHALL be required: when it is unset, or when it is only a +hostname with no org path, the swap SHALL stop with a clear error. The driver +SHALL NOT default `SWAP_REGISTRY` to `IMAGE_REGISTRY`; `IMAGE_REGISTRY` is the +baseline image prefix and SHALL NOT be the swap push destination. +`OPENSHIFT_IMAGE_REGISTRY` SHALL NOT be used. The driver SHALL NOT push to the +OpenShift internal registry (`image-registry.openshift-image-registry.svc`) +from the laptop: that hostname does not resolve outside the cluster, and many +shared clusters (ROSA/OSD) expose no public registry hostname. The swap SHALL +authenticate the container engine with `PULL_SECRET` (a +`kubernetes.io/dockerconfigjson` Secret YAML, or a raw Docker config JSON). +`KIND_PULL_SECRET` SHALL remain accepted as an alias when `PULL_SECRET` is +unset. When that file contains credentials for the `SWAP_REGISTRY` host, the +swap SHALL log in with those credentials and SHALL NOT require an interactive +`podman login`. After a successful push, the driver SHALL update the component +Deployment image refs to the pushed identity. The digest SHALL be the registry +manifest digest recorded by that push (`podman push --digestfile`, or a +`digest: sha256:...` line from a docker-style push log). The driver SHALL NOT +pin a digest from `inspect` of the local image, and SHALL NOT scrape blob or +config SHAs from push progress: those values are not the digest the registry +stores, so a cluster pull by that digest fails with manifest unknown. When no +registry digest is available, the driver SHALL pin the unique tag. + +The swap build SHALL target the OpenShift node architecture, not the laptop +architecture. When `SWAP_PLATFORM` is set (`linux/amd64` or `linux/arm64`), +the driver SHALL use that GOARCH. When it is unset, the driver SHALL read the +architecture from the cluster nodes. Go component Dockerfiles SHALL honor +`TARGETARCH` so an arm64 laptop can cross-compile an amd64 binary with +`CGO_ENABLED=0`. A native laptop build pushed to amd64 nodes SHALL NOT be +used: that fails init with `Exec format error`. Because more than one developer can share one cluster, each working-tree image SHALL have an immutable identity scoped to the source commit and to @@ -532,12 +570,42 @@ build, which run the baseline image, and the exact image each one runs. - THEN the scripts build the API server image from the working tree - AND the pushed image has an immutable identity scoped to the commit and `OPENSHIFT_NAMESPACE` -- AND the scripts push the image to a registry that the cluster can pull +- AND the scripts push the image to `${SWAP_REGISTRY}/hypershell-api-server` + (or `${SWAP_REGISTRY}/${SWAP_REPOSITORY}` when that override is set), not + `IMAGE_REGISTRY` and not the OpenShift internal registry +- AND the scripts update the API server Deployment image refs to that identity - AND the scripts record that image identity in the per-namespace swap state - AND the scripts roll out the API server deployment to the pushed image - AND `make openshift-status` reports the API server as a working-tree build with its exact image identity +#### Scenario: Swap pins the registry digest from the push + +- GIVEN the container engine's local image digest differs from the digest the + registry stored for the pushed tag +- WHEN the developer runs `make openshift-api-server-up` +- THEN the scripts pin the Deployment to the registry manifest digest recorded + by that push +- AND they SHALL NOT pin a digest from `inspect` of the local image +- AND they SHALL NOT scrape blob or config SHAs from push progress + +#### Scenario: Swap builds for the cluster architecture + +- GIVEN the developer laptop is arm64 +- AND the OpenShift nodes are amd64 +- WHEN the developer runs `make openshift-api-server-up` +- THEN the scripts build the API server with `TARGETARCH=amd64` +- AND the migrate init container SHALL start without `Exec format error` + +#### Scenario: Swap without SWAP_REGISTRY stops + +- GIVEN a HyperShell deployment exists on OpenShift with baseline images +- AND `SWAP_REGISTRY` is unset +- WHEN the developer runs `make openshift-api-server-up` +- THEN the command SHALL stop with a clear error +- AND it SHALL NOT push to `IMAGE_REGISTRY` +- AND it SHALL NOT push to the OpenShift internal registry + #### Scenario: Revert a swapped component - GIVEN the API server runs a working-tree build on OpenShift From 2d8648f174493e080f16f5ba693f7c04d32ba390 Mon Sep 17 00:00:00 2001 From: Kyle Squizzato Date: Thu, 3 Sep 2026 08:38:34 -0700 Subject: [PATCH 11/14] feat(e2e): support build-and-push OpenShift workflow from arm Signed-off-by: Kyle Squizzato --- DEVELOPMENT.md | 3 ++- Makefile | 2 +- components/api-server/Dockerfile | 16 +++++++++++--- components/cli/Dockerfile | 17 ++++++++++++--- components/control-plane/Dockerfile | 16 +++++++++++--- components/web-console/Dockerfile | 13 ++++++++++-- components/web-console/README.md | 4 +++- scripts/cluster/drivers/openshift.sh | 5 +---- scripts/cluster/lib_test.sh | 22 ++++++++++++++++++++ specs/platform/openshift-development.spec.md | 15 +++++++------ 10 files changed, 89 insertions(+), 24 deletions(-) diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md index 2f8ca2f1..51263aa9 100644 --- a/DEVELOPMENT.md +++ b/DEVELOPMENT.md @@ -452,7 +452,8 @@ Each swap builds from the working tree, pushes an immutable image (commit + namespace tag) to `SWAP_REGISTRY`, and updates the Deployment image refs to the registry digest from that push (not the local image digest). Images are built for the OpenShift node architecture (`SWAP_PLATFORM`, or detected from -the cluster), not the laptop architecture. `SWAP_REGISTRY` is the org prefix only (`quay.io/`). Repo +the cluster) via `--platform linux/`. Component Dockerfiles pin HI bases +per architecture. `SWAP_REGISTRY` is the org prefix only (`quay.io/`). Repo names default to `hypershell-api-server`, `hypershell-controller`, and `hypershell-web-console`; set `SWAP_REPOSITORY` to override the repo for the current swap. `SWAP_REGISTRY` is required; swaps do not use `IMAGE_REGISTRY`. diff --git a/Makefile b/Makefile index 491d914f..9329fbc9 100644 --- a/Makefile +++ b/Makefile @@ -129,7 +129,7 @@ help: @echo "" @echo " Test & Lint" @echo " test-all Run all test suites" - @echo " e2e Run E2E tests locally (requires Kind cluster)" + @echo " e2e Run E2E tests against target KUBECONFIG cluster" @echo " e2e-performance Run the performance harness (requires a running cluster)" @echo " e2e-performance-report Tabulate recent local performance runs" @echo " lint Run all linters (Go + JS/TS)" diff --git a/components/api-server/Dockerfile b/components/api-server/Dockerfile index 736a320e..4290e164 100644 --- a/components/api-server/Dockerfile +++ b/components/api-server/Dockerfile @@ -1,5 +1,13 @@ # syntax=docker/dockerfile:1 -FROM registry.access.redhat.com/hi/go:1.26.7@sha256:4c7c064cc73698c13384d1445ab93595f3b9309dd89ed45b9992e67284e3ed82 AS builder +# HI pins are the per-arch manifests from hi/go:1.26.7 and hi/static:1787099997. +# BUILDARCH selects a native Go toolchain; TARGETARCH selects the runtime and GOARCH. +# Bump amd64 and arm64 together when changing tags. +ARG BUILDARCH +ARG TARGETARCH + +FROM registry.access.redhat.com/hi/go:1.26.7@sha256:2a2215ce496fdd33fdc19739eaadde902f7719c8be222f22f92fd7da02c889cb AS go-amd64 +FROM registry.access.redhat.com/hi/go:1.26.7@sha256:ba850dda34c47052bb8db5561c80c43fb08d0f9a729c90fddd755a2d5b46f74b AS go-arm64 +FROM go-${BUILDARCH} AS builder WORKDIR /workspace @@ -14,12 +22,14 @@ COPY openapi/ openapi/ ARG GIT_VERSION=unknown ARG BUILD_TIME=unknown ARG TARGETARCH -RUN CGO_ENABLED=0 GOOS=linux GOARCH="${TARGETARCH:-$(go env GOARCH)}" go build -mod=mod -ldflags="-s -w \ +RUN CGO_ENABLED=0 GOOS=linux GOARCH="${TARGETARCH}" go build -mod=mod -ldflags="-s -w \ -X github.com/openshift-online/hypershell/components/api-server/pkg/api.Version=${GIT_VERSION} \ -X 'github.com/openshift-online/hypershell/components/api-server/pkg/api.BuildTime=${BUILD_TIME}'" \ -o hypershell ./cmd/hypershell -FROM registry.access.redhat.com/hi/static:1787099997@sha256:f4d5109b57cf7eab0a7adc566f2d78f80fa0c5ec9ccab698c9fb8eb448db6071 +FROM registry.access.redhat.com/hi/static:1787099997@sha256:3d43712a61ce01c7049e983e7f4c4609aaad7582c7bbd930b57372f82c446f98 AS static-amd64 +FROM registry.access.redhat.com/hi/static:1787099997@sha256:2ccd532da5e8868a92f479a7268a2fce65279c1f7c8a8dd0880d399c04d6a356 AS static-arm64 +FROM static-${TARGETARCH} COPY --from=builder /workspace/hypershell /usr/local/bin/ diff --git a/components/cli/Dockerfile b/components/cli/Dockerfile index 6bcf5e99..55b77239 100644 --- a/components/cli/Dockerfile +++ b/components/cli/Dockerfile @@ -1,5 +1,13 @@ # syntax=docker/dockerfile:1 -FROM registry.access.redhat.com/hi/go:1.26.7@sha256:4c7c064cc73698c13384d1445ab93595f3b9309dd89ed45b9992e67284e3ed82 AS builder +# HI pins are the per-arch manifests from hi/go:1.26.7 and hi/static:1787099997. +# BUILDARCH selects a native Go toolchain; TARGETARCH selects the runtime and GOARCH. +# Bump amd64 and arm64 together when changing tags. +ARG BUILDARCH +ARG TARGETARCH + +FROM registry.access.redhat.com/hi/go:1.26.7@sha256:2a2215ce496fdd33fdc19739eaadde902f7719c8be222f22f92fd7da02c889cb AS go-amd64 +FROM registry.access.redhat.com/hi/go:1.26.7@sha256:ba850dda34c47052bb8db5561c80c43fb08d0f9a729c90fddd755a2d5b46f74b AS go-arm64 +FROM go-${BUILDARCH} AS builder WORKDIR /workspace @@ -9,9 +17,12 @@ RUN go mod download COPY cmd/ cmd/ COPY pkg/ pkg/ -RUN CGO_ENABLED=0 go build -ldflags="-s -w" -o hsctl ./cmd/hypershell +ARG TARGETARCH +RUN CGO_ENABLED=0 GOOS=linux GOARCH="${TARGETARCH}" go build -ldflags="-s -w" -o hsctl ./cmd/hypershell -FROM registry.access.redhat.com/hi/static:1787099997@sha256:f4d5109b57cf7eab0a7adc566f2d78f80fa0c5ec9ccab698c9fb8eb448db6071 +FROM registry.access.redhat.com/hi/static:1787099997@sha256:3d43712a61ce01c7049e983e7f4c4609aaad7582c7bbd930b57372f82c446f98 AS static-amd64 +FROM registry.access.redhat.com/hi/static:1787099997@sha256:2ccd532da5e8868a92f479a7268a2fce65279c1f7c8a8dd0880d399c04d6a356 AS static-arm64 +FROM static-${TARGETARCH} COPY --from=builder /workspace/hsctl /usr/local/bin/ diff --git a/components/control-plane/Dockerfile b/components/control-plane/Dockerfile index 210d3e45..c1a0e180 100644 --- a/components/control-plane/Dockerfile +++ b/components/control-plane/Dockerfile @@ -1,5 +1,13 @@ # syntax=docker/dockerfile:1 -FROM registry.access.redhat.com/hi/go:1.26.7@sha256:4c7c064cc73698c13384d1445ab93595f3b9309dd89ed45b9992e67284e3ed82 AS builder +# HI pins are the per-arch manifests from hi/go:1.26.7 and hi/static:1787099997. +# BUILDARCH selects a native Go toolchain; TARGETARCH selects the runtime and GOARCH. +# Bump amd64 and arm64 together when changing tags. +ARG BUILDARCH +ARG TARGETARCH + +FROM registry.access.redhat.com/hi/go:1.26.7@sha256:2a2215ce496fdd33fdc19739eaadde902f7719c8be222f22f92fd7da02c889cb AS go-amd64 +FROM registry.access.redhat.com/hi/go:1.26.7@sha256:ba850dda34c47052bb8db5561c80c43fb08d0f9a729c90fddd755a2d5b46f74b AS go-arm64 +FROM go-${BUILDARCH} AS builder WORKDIR /workspace @@ -17,9 +25,11 @@ COPY components/control-plane/cmd/ components/control-plane/cmd/ COPY components/control-plane/internal/ components/control-plane/internal/ ARG TARGETARCH -RUN cd components/control-plane && CGO_ENABLED=0 GOOS=linux GOARCH="${TARGETARCH:-$(go env GOARCH)}" go build -mod=mod -ldflags="-s -w" -o /workspace/hypershell-controller ./cmd/hypershell-controller +RUN cd components/control-plane && CGO_ENABLED=0 GOOS=linux GOARCH="${TARGETARCH}" go build -mod=mod -ldflags="-s -w" -o /workspace/hypershell-controller ./cmd/hypershell-controller -FROM registry.access.redhat.com/hi/static:1787099997@sha256:f4d5109b57cf7eab0a7adc566f2d78f80fa0c5ec9ccab698c9fb8eb448db6071 +FROM registry.access.redhat.com/hi/static:1787099997@sha256:3d43712a61ce01c7049e983e7f4c4609aaad7582c7bbd930b57372f82c446f98 AS static-amd64 +FROM registry.access.redhat.com/hi/static:1787099997@sha256:2ccd532da5e8868a92f479a7268a2fce65279c1f7c8a8dd0880d399c04d6a356 AS static-arm64 +FROM static-${TARGETARCH} COPY --from=builder /workspace/hypershell-controller /usr/local/bin/ COPY components/control-plane/manifests/ /manifests/ diff --git a/components/web-console/Dockerfile b/components/web-console/Dockerfile index 52f9c821..35e533f0 100644 --- a/components/web-console/Dockerfile +++ b/components/web-console/Dockerfile @@ -1,5 +1,12 @@ # syntax=docker/dockerfile:1 -FROM registry.access.redhat.com/hi/nodejs:24.18.1-builder@sha256:e2a04f7443db3d27751ed6e9d74ee5c3a02217dbf6c8769e468ef3b1b2a3e6fa AS build +# HI pins are the per-arch manifests from hi/nodejs:24.18.1-builder and hi/nodejs:24.18.1. +# TARGETARCH selects both stages so native addons and the node binary match the cluster. +# Bump amd64 and arm64 together when changing tags. +ARG TARGETARCH + +FROM registry.access.redhat.com/hi/nodejs:24.18.1-builder@sha256:7d3b87dbb6bbf8fc85c865f8a00c5355961dae0e0e4ead15d530af3c8d6e3ebe AS build-amd64 +FROM registry.access.redhat.com/hi/nodejs:24.18.1-builder@sha256:5034714f7a4bdd7423231f880b0863b8e69c5aa9b8f5ab69ed1cd1e76760f0e0 AS build-arm64 +FROM build-${TARGETARCH} AS build USER root @@ -28,7 +35,9 @@ RUN pnpm run build:web \ && pnpm --filter @openshift-online/hypershell-web-console-bff deploy --prod /tmp/web-console \ && cp -R components/web-console/build/client /tmp/web-console/public -FROM registry.access.redhat.com/hi/nodejs:24.18.1@sha256:07b0f6cf5dabef30b1efa030eea020095b1ea227c088dbcbe0ebd35bb24b7ad9 +FROM registry.access.redhat.com/hi/nodejs:24.18.1@sha256:6725f3730de000b6255b97d31e2ef53916f35397b1fb779bdb8d4a3a1c84ad50 AS runtime-amd64 +FROM registry.access.redhat.com/hi/nodejs:24.18.1@sha256:8b3a067f10336acb6798f4d8251882fed107c90116b69a469e02472ecc1c7648 AS runtime-arm64 +FROM runtime-${TARGETARCH} LABEL org.opencontainers.image.title="HyperShell web console" \ org.opencontainers.image.description="HyperShell web console BFF and static application" \ diff --git a/components/web-console/README.md b/components/web-console/README.md index 7fa9ece6..582a2a92 100644 --- a/components/web-console/README.md +++ b/components/web-console/README.md @@ -66,6 +66,8 @@ podman run --rm \ localhost/hypershell-web-console:dev ``` -The image is multi-stage and pins Red Hat's `hi/nodejs` builder and runtime variants by digest. Its runtime contains the production BFF dependency closure and built assets, runs as a numeric non-root user, writes no files, exposes port 8080, and provides `/health/live` and `/health/ready` probes. +The image is multi-stage and pins Red Hat's `hi/nodejs` builder and runtime +variants by per-architecture digest (`amd64` and `arm64`). `docker`/`podman` +`--platform` selects the pin. Its runtime contains the production BFF dependency closure and built assets, runs as a numeric non-root user, writes no files, exposes port 8080, and provides `/health/live` and `/health/ready` probes. The BFF proxies `/api/*` to the fixed `HYPERSHELL_API_ORIGIN` origin, which defaults to `http://127.0.0.1:8000` for a colocated API. Set that variable to an HTTP(S) origin reachable from the container when the API is deployed separately. Browser cookies and authorization headers are not forwarded; the current product assumption is that all gateway records are visible. `HYPERSHELL_API_TIMEOUT_MS` bounds each upstream request and defaults to 30 seconds. diff --git a/scripts/cluster/drivers/openshift.sh b/scripts/cluster/drivers/openshift.sh index 18721816..9e607fa3 100755 --- a/scripts/cluster/drivers/openshift.sh +++ b/scripts/cluster/drivers/openshift.sh @@ -1309,10 +1309,7 @@ push_component_image() { target_arch="$(swap_target_goarch)" || exit 1 info "Building ${component} from working tree for linux/${target_arch}..." - if [[ "${component}" == "web-console" ]]; then - warn "web-console cannot Go-cross-compile; this Node image matches the laptop unless you rebuild on ${target_arch}." - fi - ${CONTAINER_ENGINE} build -t "${LOCAL_IMAGE}" \ + ${CONTAINER_ENGINE} build --platform "linux/${target_arch}" -t "${LOCAL_IMAGE}" \ -f "${REPO_ROOT}/${DOCKERFILE}" ${BUILD_ARGS[@]+"${BUILD_ARGS[@]}"} \ --build-arg "TARGETARCH=${target_arch}" \ --build-arg "TARGETOS=linux" \ diff --git a/scripts/cluster/lib_test.sh b/scripts/cluster/lib_test.sh index b512c587..6410bef0 100755 --- a/scripts/cluster/lib_test.sh +++ b/scripts/cluster/lib_test.sh @@ -859,6 +859,28 @@ else FAIL=$((FAIL + 1)) echo 'FAIL: swap build does not pass TARGETARCH for the cluster node architecture' fi +if grep -A80 '^push_component_image()' "${SCRIPT_DIR}/drivers/openshift.sh" \ + | grep -q -- '--platform'; then + PASS=$((PASS + 1)) +else + FAIL=$((FAIL + 1)) + echo 'FAIL: swap build does not pass --platform linux/' +fi +if grep -q 'AS go-amd64' "${REPO_ROOT}/components/api-server/Dockerfile" \ + && grep -q 'AS go-arm64' "${REPO_ROOT}/components/api-server/Dockerfile" \ + && grep -q 'AS static-amd64' "${REPO_ROOT}/components/api-server/Dockerfile" \ + && grep -q 'AS static-arm64' "${REPO_ROOT}/components/api-server/Dockerfile" \ + && grep -q 'AS go-amd64' "${REPO_ROOT}/components/control-plane/Dockerfile" \ + && grep -q 'AS go-arm64' "${REPO_ROOT}/components/control-plane/Dockerfile" \ + && grep -q 'AS build-amd64' "${REPO_ROOT}/components/web-console/Dockerfile" \ + && grep -q 'AS build-arm64' "${REPO_ROOT}/components/web-console/Dockerfile" \ + && grep -q 'AS runtime-amd64' "${REPO_ROOT}/components/web-console/Dockerfile" \ + && grep -q 'AS runtime-arm64' "${REPO_ROOT}/components/web-console/Dockerfile"; then + PASS=$((PASS + 1)) +else + FAIL=$((FAIL + 1)) + echo 'FAIL: swap Dockerfiles do not pin HI bases per amd64 and arm64' +fi if grep -q 'GOARCH="${TARGETARCH' "${REPO_ROOT}/components/api-server/Dockerfile" \ && grep -q 'GOARCH="${TARGETARCH' "${REPO_ROOT}/components/control-plane/Dockerfile"; then PASS=$((PASS + 1)) diff --git a/specs/platform/openshift-development.spec.md b/specs/platform/openshift-development.spec.md index a13a57fc..90bdac2f 100644 --- a/specs/platform/openshift-development.spec.md +++ b/specs/platform/openshift-development.spec.md @@ -544,11 +544,13 @@ registry digest is available, the driver SHALL pin the unique tag. The swap build SHALL target the OpenShift node architecture, not the laptop architecture. When `SWAP_PLATFORM` is set (`linux/amd64` or `linux/arm64`), -the driver SHALL use that GOARCH. When it is unset, the driver SHALL read the -architecture from the cluster nodes. Go component Dockerfiles SHALL honor -`TARGETARCH` so an arm64 laptop can cross-compile an amd64 binary with -`CGO_ENABLED=0`. A native laptop build pushed to amd64 nodes SHALL NOT be -used: that fails init with `Exec format error`. +the driver SHALL use that architecture. When it is unset, the driver SHALL read +the architecture from the cluster nodes. The driver SHALL pass +`--platform linux/` to the container build. Component Dockerfiles SHALL +pin Red Hat Hardened Image manifests per architecture (`amd64` and `arm64`) +and SHALL select the pin with `TARGETARCH` (and `BUILDARCH` for a native Go +toolchain). A single-arch pin SHALL NOT be used: that produces `Exec format +error` when an arm64 laptop image is pulled by amd64 nodes. Because more than one developer can share one cluster, each working-tree image SHALL have an immutable identity scoped to the source commit and to @@ -594,7 +596,8 @@ build, which run the baseline image, and the exact image each one runs. - GIVEN the developer laptop is arm64 - AND the OpenShift nodes are amd64 - WHEN the developer runs `make openshift-api-server-up` -- THEN the scripts build the API server with `TARGETARCH=amd64` +- THEN the scripts build the API server with `--platform linux/amd64` +- AND the Dockerfiles select the amd64 HI digest pins - AND the migrate init container SHALL start without `Exec format error` #### Scenario: Swap without SWAP_REGISTRY stops From 21e3d0e4d2025a991c726d88ebb4df80b6d152d4 Mon Sep 17 00:00:00 2001 From: Kyle Squizzato Date: Thu, 3 Sep 2026 09:03:18 -0700 Subject: [PATCH 12/14] fix(ci): update check to validate multi-stage pins Assisted-by: Claude Sonnet 5 Signed-off-by: Kyle Squizzato --- scripts/check_dependency_pins.py | 40 ++++++++++++++++++++++++++- scripts/test_check_dependency_pins.py | 27 ++++++++++++++++++ 2 files changed, 66 insertions(+), 1 deletion(-) diff --git a/scripts/check_dependency_pins.py b/scripts/check_dependency_pins.py index de4d3894..c589c148 100755 --- a/scripts/check_dependency_pins.py +++ b/scripts/check_dependency_pins.py @@ -22,6 +22,8 @@ _FROM_LINE = re.compile( r"^\s*FROM\s+(?:--[^\s]+\s+)*([^\s]+)", re.IGNORECASE ) +_AS_CLAUSE = re.compile(r"\bAS\s+([^\s]+)", re.IGNORECASE) +_TEMPLATE_TOKEN = re.compile(r"\$\{[^}]+\}|\$[A-Za-z_][A-Za-z0-9_]*") _IMAGE_LINE = re.compile(r"^(\s*)image:\s*([^\s#]+)") _MAKE_IMAGE = re.compile( r"^\s*([A-Z][A-Z0-9_]*_IMAGE)\s*(?::|\?)?=\s*([^\s#]+)" @@ -170,10 +172,37 @@ def _workflow_violations( return violations +def _stage_reference_pattern(reference: str) -> re.Pattern[str]: + """Build a regex matching a FROM reference that interpolates build ARGs. + + `FROM go-${BUILDARCH}` should match a stage declared as `AS go-amd64` + without knowing the ARG's value, so each template token becomes a + non-empty wildcard. + """ + pattern_parts = [] + last_end = 0 + for token in _TEMPLATE_TOKEN.finditer(reference): + pattern_parts.append(re.escape(reference[last_end : token.start()])) + pattern_parts.append(r".+") + last_end = token.end() + pattern_parts.append(re.escape(reference[last_end:])) + return re.compile("^" + "".join(pattern_parts) + "$") + + def _dockerfile_violations( relative_path: str, lines: list[str] ) -> list[tuple[str, int, str]]: violations = [] + stage_names: set[str] = set() + for line in lines: + if line.lstrip().startswith("#"): + continue + if not _FROM_LINE.match(line): + continue + as_match = _AS_CLAUSE.search(line) + if as_match: + stage_names.add(_unquote(as_match.group(1))) + for line_number, line in enumerate(lines, start=1): if line.lstrip().startswith("#"): continue @@ -189,7 +218,16 @@ def _dockerfile_violations( if not match: continue reference = _unquote(match.group(1)) - if reference.lower() != "scratch" and not _is_digest_pinned(reference): + if reference.lower() == "scratch" or reference in stage_names: + continue + # A FROM referencing an earlier pinned stage by name (e.g. per-arch + # variants selected via `FROM go-${BUILDARCH}`) needs no digest of + # its own: the stage it resolves to was already validated above. + if _TEMPLATE_TOKEN.search(reference) and any( + _stage_reference_pattern(reference).match(name) for name in stage_names + ): + continue + if not _is_digest_pinned(reference): violations.append( (relative_path, line_number, "base image lacks a sha256 digest") ) diff --git a/scripts/test_check_dependency_pins.py b/scripts/test_check_dependency_pins.py index 187ace20..9d350f15 100644 --- a/scripts/test_check_dependency_pins.py +++ b/scripts/test_check_dependency_pins.py @@ -75,6 +75,33 @@ def test_allows_digest_pinned_external_image(self): ), ) + def test_allows_from_selecting_a_digest_pinned_stage_by_build_arg(self): + self.assertEqual( + [], + CHECKER._dockerfile_violations( + "components/example/Dockerfile", + [ + "ARG BUILDARCH", + "FROM registry.access.redhat.com/hi/go:1@sha256:" + + "a" * 64 + + " AS go-amd64", + "FROM registry.access.redhat.com/hi/go:1@sha256:" + + "b" * 64 + + " AS go-arm64", + "FROM go-${BUILDARCH} AS builder", + ], + ), + ) + + def test_rejects_from_selecting_an_undeclared_stage(self): + self.assertEqual( + [("components/example/Dockerfile", 1, "base image lacks a sha256 digest")], + CHECKER._dockerfile_violations( + "components/example/Dockerfile", + ["FROM go-${BUILDARCH} AS builder"], + ), + ) + if __name__ == "__main__": unittest.main() From 48ec5de73fc0a305ed51ca3a419347c45762528d Mon Sep 17 00:00:00 2001 From: Kyle Squizzato Date: Thu, 3 Sep 2026 09:11:45 -0700 Subject: [PATCH 13/14] fix(e2e-openshift): refuse unowned project deletion Fail closed during teardown so the lifecycle cannot delete an unrelated selected OpenShift project. Assisted-by: GPT-5.6 Terra Signed-off-by: Kyle Squizzato --- scripts/cluster/drivers/openshift.sh | 14 +++++++--- scripts/cluster/lib_test.sh | 38 ++++++++++++++++++++++++++++ specs/platform/e2e-testing.spec.md | 2 +- tests/e2e/drivers/openshift.sh | 14 +++++++--- tests/e2e/openshift_driver_test.sh | 8 ++++-- 5 files changed, 66 insertions(+), 10 deletions(-) diff --git a/scripts/cluster/drivers/openshift.sh b/scripts/cluster/drivers/openshift.sh index 9e607fa3..6731bbaf 100755 --- a/scripts/cluster/drivers/openshift.sh +++ b/scripts/cluster/drivers/openshift.sh @@ -1074,13 +1074,19 @@ verify_owned_namespace() { local owned env_id owned="$(namespace_label_value "${ns}" "${OWNED_LABEL}")" env_id="$(namespace_label_value "${ns}" "${ENV_LABEL}")" - if [[ "${owned}" == "true" && -n "${env_id}" && -n "${OPENSHIFT_ENVIRONMENT_ID:-}" && "${env_id}" != "${OPENSHIFT_ENVIRONMENT_ID}" ]]; then - error "Namespace '${ns}' belongs to environment '${env_id}', not '${OPENSHIFT_ENVIRONMENT_ID}'. Refusing to delete it." + + # Destructive operations must be fail-closed. An unlabeled project may be a + # developer's currently selected project, but down cannot prove that this + # lifecycle created it; deleting it could erase unrelated workloads. + if [[ "${owned}" != "true" || -z "${env_id}" ]]; then + error "Namespace '${ns}' is not a complete HyperShell environment (expected ${OWNED_LABEL}=true and ${ENV_LABEL}). Refusing to delete it." exit 1 fi - if [[ "${owned}" == "true" && -n "${env_id}" ]]; then - OPENSHIFT_ENVIRONMENT_ID="${env_id}" + if [[ -n "${OPENSHIFT_ENVIRONMENT_ID:-}" && "${env_id}" != "${OPENSHIFT_ENVIRONMENT_ID}" ]]; then + error "Namespace '${ns}' belongs to environment '${env_id}', not '${OPENSHIFT_ENVIRONMENT_ID}'. Refusing to delete it." + exit 1 fi + OPENSHIFT_ENVIRONMENT_ID="${env_id}" } delete_hypershell_resources() { diff --git a/scripts/cluster/lib_test.sh b/scripts/cluster/lib_test.sh index 6410bef0..2a30596c 100755 --- a/scripts/cluster/lib_test.sh +++ b/scripts/cluster/lib_test.sh @@ -40,6 +40,22 @@ assert_fail() { fi } +# Exercise the destructive-path ownership predicate with a minimal OpenShift +# seam. The driver exits on an unsafe namespace, so this wrapper is a subshell. +verify_openshift_namespace_fixture() ( + # shellcheck source=drivers/openshift.sh + source "${SCRIPT_DIR}/drivers/openshift.sh" + namespace_exists() { return 0; } + namespace_label_value() { + case "$2" in + "${OWNED_LABEL}") printf '%s' "${FIXTURE_OWNED:-}" ;; + "${ENV_LABEL}") printf '%s' "${FIXTURE_ENV_ID:-}" ;; + esac + } + OPENSHIFT_ENVIRONMENT_ID="${FIXTURE_EXPECTED_ENV_ID:-}" + verify_owned_namespace fixture +) + # --- RFC 1123 / sanitization --- assert_eq "kube-admin" "$(sanitize_dns_label 'kube:admin')" "sanitize kube:admin" assert_eq "user-redhat-com" "$(sanitize_dns_label 'user@redhat.com')" "sanitize email" @@ -150,6 +166,28 @@ if grep -E 'deletion started|removal started' "${SCRIPT_DIR}/drivers/openshift.s else PASS=$((PASS + 1)) fi +if grep -A30 '^verify_owned_namespace()' "${SCRIPT_DIR}/drivers/openshift.sh" \ + | grep -q '"${owned}" != "true" || -z "${env_id}"'; then + PASS=$((PASS + 1)) +else + FAIL=$((FAIL + 1)) + echo 'FAIL: OpenShift down does not refuse unlabeled or partially labeled namespaces' +fi +if grep -A30 '^verify_owned_namespace()' "${SCRIPT_DIR}/drivers/openshift.sh" \ + | grep -q 'Refusing to delete it'; then + PASS=$((PASS + 1)) +else + FAIL=$((FAIL + 1)) + echo 'FAIL: OpenShift down ownership verification does not fail closed' +fi +FIXTURE_OWNED= FIXTURE_ENV_ID= \ + assert_fail "unlabeled project is refused by openshift-down" verify_openshift_namespace_fixture +FIXTURE_OWNED=true FIXTURE_ENV_ID= \ + assert_fail "partially labeled project is refused by openshift-down" verify_openshift_namespace_fixture +FIXTURE_OWNED=true FIXTURE_ENV_ID=environment-a FIXTURE_EXPECTED_ENV_ID=environment-b \ + assert_fail "foreign environment is refused by openshift-down" verify_openshift_namespace_fixture +FIXTURE_OWNED=true FIXTURE_ENV_ID=environment-a FIXTURE_EXPECTED_ENV_ID=environment-a \ + assert_ok "owned environment is accepted by openshift-down" verify_openshift_namespace_fixture if grep -A50 '^cluster_down()' "${SCRIPT_DIR}/drivers/openshift.sh" | grep -q 'OPENSHIFT_KEYCLOAK_NAMESPACE'; then PASS=$((PASS + 1)) else diff --git a/specs/platform/e2e-testing.spec.md b/specs/platform/e2e-testing.spec.md index 16ffb841..7297ebae 100644 --- a/specs/platform/e2e-testing.spec.md +++ b/specs/platform/e2e-testing.spec.md @@ -222,7 +222,7 @@ Each target SHALL auto-detect the driver from the current KUBECONFIG context (se **Preconditions (owned by `openshift-development.spec.md`).** These runs assume HyperShell is already deployed on the cluster through `make openshift-up` (`kustomize build deploy/openshift/` mapped into the current `oc` project, or `OPENSHIFT_NAMESPACE`). That bring-up creates the companion `${OPENSHIFT_NAMESPACE}-keycloak` project, applies Routes for the API, web console, and Keycloak, applies `keycloak-allow-platform` so platform pods can reach JWKS, applies per-environment ClusterRoles and ClusterRoleBindings named `${OPENSHIFT_NAMESPACE}-dev-*`, and applies the privileged SCC RoleBinding `hypershell-sandbox-scc`. The cluster infrastructure bootstrap (shared Gateway, GatewayClass, certificate issuer, wildcard certificate) is in place per `openshift-development.spec.md`. The suite SHALL fail with a clear error, not a broken run, when the API Route or the gateway infrastructure is absent. -**Driver behavior needed for parity.** For the shared suite to pass on OpenShift, the OpenShift driver SHALL derive the OIDC issuer from the Keycloak Route in `${OPENSHIFT_NAMESPACE}-keycloak` (not the Kind default `keycloak.hypershell.localhost`), SHALL return `get_cluster_domain` from the same shared-Gateway listener hostname `make openshift-up` used, and SHALL provide the same Keycloak admin and role-assignment helpers the Kind driver provides, so the RBAC areas (developer and platform-admin) run unchanged. The OpenShift deployment SHALL enforce RBAC (`RBAC_ENFORCE=true`) and SHALL keep the OpenShift SCC posture (per-namespace privileged SCC for sandbox pods), so the sandbox and RBAC areas behave the same as on Kind. These behaviors are specified in `openshift-development.spec.md`; this spec only depends on them. +**Driver behavior needed for parity.** For the shared suite to pass on OpenShift, the OpenShift driver SHALL use the current `oc` project when `OPENSHIFT_NAMESPACE` is unset (and fail clearly when neither is available), matching `make openshift-up`; derive the OIDC issuer from the Keycloak Route in `${OPENSHIFT_NAMESPACE}-keycloak` (not the Kind default `keycloak.hypershell.localhost`); return `get_cluster_domain` from the same shared-Gateway listener hostname `make openshift-up` used; and provide the same Keycloak admin and role-assignment helpers the Kind driver provides, so the RBAC areas (developer and platform-admin) run unchanged. The OpenShift deployment SHALL enforce RBAC (`RBAC_ENFORCE=true`) and SHALL keep the OpenShift SCC posture (per-namespace privileged SCC for sandbox pods), so the sandbox and RBAC areas behave the same as on Kind. These behaviors are specified in `openshift-development.spec.md`; this spec only depends on them. **Namespace GC timing.** Area 11 exercises the periodic namespace reaper. To make it pass on OpenShift without waiting the production GC defaults (5m sweep / 10m grace), the OpenShift deployment SHOULD set shortened `GATEWAY_NAMESPACE_GC_INTERVAL` and `GATEWAY_NAMESPACE_GC_GRACE_PERIOD` (as the Kind overlay does), or the user SHOULD raise `E2E_ORPHAN_GC_TIMEOUT` and `E2E_GC_TIMEOUT` to fit the cluster's configured timing. diff --git a/tests/e2e/drivers/openshift.sh b/tests/e2e/drivers/openshift.sh index 915742d4..b648a40b 100644 --- a/tests/e2e/drivers/openshift.sh +++ b/tests/e2e/drivers/openshift.sh @@ -2,7 +2,8 @@ # OpenShift infrastructure driver for the shared e2e and performance suites. # # The environment must already exist. OPENSHIFT_NAMESPACE selects its platform -# namespace; other runtime settings are discovered from that deployment. +# namespace; when unset, use the current oc project just like openshift-up. +# Other runtime settings are discovered from that deployment. # Reuse the infrastructure-neutral OIDC, Keycloak role, and JWT helpers. Every # infrastructure operation and the TLS policy are overridden below. @@ -15,8 +16,15 @@ source "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/kind.sh" _openshift_require_config() { if [[ -z "${OPENSHIFT_NAMESPACE:-}" ]]; then - red " OPENSHIFT_NAMESPACE is required for an OpenShift e2e run" - return 1 + local project + project="$(oc project -q 2>/dev/null || true)" + if [[ -z "${project}" ]]; then + red " OPENSHIFT_NAMESPACE is unset and no oc project is selected" + red " Run 'oc project ' or set OPENSHIFT_NAMESPACE before running make e2e" + return 1 + fi + OPENSHIFT_NAMESPACE="${project}" + dim " OPENSHIFT_NAMESPACE unset; using oc project '${OPENSHIFT_NAMESPACE}'" fi E2E_HS_NAMESPACE="${OPENSHIFT_NAMESPACE}" E2E_KEYCLOAK_NAMESPACE="${OPENSHIFT_NAMESPACE}-keycloak" diff --git a/tests/e2e/openshift_driver_test.sh b/tests/e2e/openshift_driver_test.sh index 477a706b..1e0828ea 100644 --- a/tests/e2e/openshift_driver_test.sh +++ b/tests/e2e/openshift_driver_test.sh @@ -25,6 +25,7 @@ OPENSHIFT_NAMESPACE=test-team oc() { local args="$*" case "$args" in + "project -q") printf '%s' "${OC_PROJECT:-}" ;; *"get route hypershell-api -n test-team"*) printf '%s' 'api-test.apps.example.com' ;; *"get route keycloak -n test-team-keycloak"*) printf '%s' 'sso-test.apps.example.com' ;; *"get deployment hypershell-controller -n test-team"*) printf '%s' 'gw.test.example.com' ;; @@ -64,9 +65,12 @@ else PASS=$((PASS + 1)) fi -if (unset OPENSHIFT_NAMESPACE; discover_api_host >/dev/null 2>&1); then +resolved_namespace="$(unset OPENSHIFT_NAMESPACE; OC_PROJECT=current-project; _openshift_require_config >/dev/null; printf '%s' "${OPENSHIFT_NAMESPACE}")" +assert_eq 'current-project' "${resolved_namespace}" 'current oc project selects OpenShift E2E namespace' + +if (unset OPENSHIFT_NAMESPACE; OC_PROJECT=; _openshift_require_config >/dev/null 2>&1); then FAIL=$((FAIL + 1)) - echo 'FAIL: missing OPENSHIFT_NAMESPACE was accepted' + echo 'FAIL: missing OPENSHIFT_NAMESPACE and oc project were accepted' else PASS=$((PASS + 1)) fi From 869021d8bcca42b38413225b5f1c4ed06bdf6d24 Mon Sep 17 00:00:00 2001 From: Kyle Squizzato Date: Thu, 3 Sep 2026 13:38:53 -0700 Subject: [PATCH 14/14] fix(control-plane): backfill instance labels on legacy gateway namespaces The namespace GC selector requires hypershell.redhat.io/instance, but a live gateway in a steady-state phase (Running/Provisioning/Degraded) is skipped by the reconciler's phase gate on restart, so EnsureManagedNamespace never runs and its legacy namespace never gains the label. Add a one-shot startup backfill, driven from this instance's Gateway inventory, that stamps the instance label onto namespaces it already owns -- never creating a namespace, never touching one lacking both management labels, and never overwriting a foreign instance's label, so it stays safe on a shared cluster. Assisted-by: Claude Sonnet 5 Signed-off-by: Kyle Squizzato --- .../cmd/hypershell-controller/main.go | 54 +++--- .../internal/gateway/namespace.go | 51 ++++++ .../internal/gateway/namespace_test.go | 94 +++++++++++ .../internal/reconciler/backfill.go | 97 +++++++++++ .../internal/reconciler/backfill_test.go | 159 ++++++++++++++++++ .../internal/reconciler/namespace.go | 12 +- .../internal/supervisor/supervisor.go | 44 +++++ .../internal/supervisor/supervisor_test.go | 33 ++++ deploy/base/controller.yaml | 4 + .../openshell-gateway-namespace-gc.spec.md | 62 ++++++- 10 files changed, 566 insertions(+), 44 deletions(-) create mode 100644 components/control-plane/internal/reconciler/backfill.go create mode 100644 components/control-plane/internal/reconciler/backfill_test.go create mode 100644 components/control-plane/internal/supervisor/supervisor.go create mode 100644 components/control-plane/internal/supervisor/supervisor_test.go diff --git a/components/control-plane/cmd/hypershell-controller/main.go b/components/control-plane/cmd/hypershell-controller/main.go index 2af9a20a..ecb4e69e 100644 --- a/components/control-plane/cmd/hypershell-controller/main.go +++ b/components/control-plane/cmd/hypershell-controller/main.go @@ -26,6 +26,7 @@ import ( "github.com/openshift-online/hypershell/components/control-plane/internal/reconciler" "github.com/openshift-online/hypershell/components/control-plane/internal/serviceaccountkeycloak" "github.com/openshift-online/hypershell/components/control-plane/internal/serviceaccountprovisioner" + "github.com/openshift-online/hypershell/components/control-plane/internal/supervisor" "github.com/openshift-online/hypershell/components/control-plane/internal/watcher" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" @@ -34,43 +35,15 @@ import ( const defaultManifestsDir = "/manifests/gateway" +// instanceLabelBackfillTimeout bounds the one-shot startup backfill that stamps +// this instance's identity label onto its legacy gateway namespaces, so a stalled +// API server or apiserver cannot delay the GC reconciler's launch indefinitely. +const instanceLabelBackfillTimeout = 2 * time.Minute + func managedDatabaseWatchEligible(clientset *kubernetes.Clientset, dynamicClient dynamic.Interface) bool { return clientset != nil && dynamicClient != nil } -// supervisedRestartCap bounds the backoff between restarts of a supervised -// background component, matching the cap watchLoop uses for stream -// reconnects. -const supervisedRestartCap = 30 * time.Second - -// runSupervised runs fn in a loop so a failure in one background component -// (a watch stream, a reconciler's Run loop, the service-account provisioner) -// cannot take down the others by exiting the whole process. Every fn here is -// expected to block until ctx is done and return ctx.Err() at that point, -// exactly as watchLoop and the reconciler Run methods already do; a return -// before then is treated as a failure of that component alone; it is retried -// with the same capped exponential backoff watchLoop uses for stream -// reconnects rather than propagated to the process. -func runSupervised(ctx context.Context, name string, fn func(context.Context) error) { - backoff := time.Second - for { - err := fn(ctx) - if err == nil || ctx.Err() != nil { - return - } - log.Printf("WARN %s exited with error, restarting in %s: %v", name, backoff, err) - select { - case <-ctx.Done(): - return - case <-time.After(backoff): - } - backoff *= 2 - if backoff > supervisedRestartCap { - backoff = supervisedRestartCap - } - } -} - func main() { cfg, err := config.Load() if err != nil { @@ -270,7 +243,7 @@ func main() { watchCount++ } - // Each background component below runs under runSupervised on its own + // Each background component below runs under supervisor.Run on its own // goroutine: a failure in one (a dropped watch stream, a reconciler's Run // loop returning, the service-account provisioner's listener dying) is // logged and retried in place, and never takes the others down with it. @@ -279,7 +252,7 @@ func main() { var wg sync.WaitGroup supervise := func(name string, fn func(context.Context) error) { wg.Go(func() { - runSupervised(ctx, name, fn) + supervisor.Run(ctx, name, fn) }) } @@ -349,6 +322,17 @@ func main() { // while the control plane was down, or a gateway that failed to bootstrap). // It requires an in-cluster Kubernetes client. if clientset != nil && cfg.NamespaceGCEnabled { + // Before the first sweep, backfill this instance's identity label onto the + // legacy gateway namespaces it still owns per its API server. A Gateway in + // a steady-state phase is skipped by the reconciler's phase gate on start, + // so its pre-label namespace is otherwise never migrated and a later + // missed-delete would leak it past the instance-scoped sweep. This runs + // synchronously so the first sweep sees the freshly-labeled namespaces; it + // is best-effort and never blocks startup on failure. + backfillCtx, cancelBackfill := context.WithTimeout(ctx, instanceLabelBackfillTimeout) + reconciler.RunInstanceLabelBackfill(backfillCtx, clientset, conn, cfg.Namespace) + cancelBackfill() + gcReconciler := reconciler.NewNamespaceGCReconciler( clientset, conn, cfg.NamespaceGCInterval, cfg.NamespaceGCGracePeriod, cfg.Namespace, ) diff --git a/components/control-plane/internal/gateway/namespace.go b/components/control-plane/internal/gateway/namespace.go index 36819f22..ce9cbc32 100644 --- a/components/control-plane/internal/gateway/namespace.go +++ b/components/control-plane/internal/gateway/namespace.go @@ -209,6 +209,57 @@ func EnsureManagedNamespace(ctx context.Context, client kubernetes.Interface, na return nil } +// BackfillInstanceLabel stamps this instance's identity label onto an existing +// managed gateway namespace that predates the label, so periodic GC -- which +// selects on hypershell.redhat.io/instance -- can observe it once its Gateway is +// deleted. It is the startup reclaim path for legacy namespaces this instance +// still owns per its API server, and is deliberately narrower than +// EnsureManagedNamespace so it is safe to run against a shared cluster: +// +// - It never creates a namespace. An absent namespace is a no-op success; the +// gateway reconciler creates it when the Gateway is next reconciled. +// - It only touches namespaces that already carry BOTH management labels, i.e. +// namespaces this control plane created. A namespace lacking them is left +// untouched even if a Gateway records its name. +// - It never overwrites a foreign instance label; a namespace already claimed +// by another HyperShell is left untouched. +// +// The label is applied with a merge patch so a concurrent reconcile writing other +// labels is not clobbered. It returns labeled=true only when it stamped the label. +func BackfillInstanceLabel(ctx context.Context, client kubernetes.Interface, namespace, instance string) (bool, error) { + if instance == "" { + return false, fmt.Errorf("refusing to backfill namespace %s without a control-plane instance identity", namespace) + } + ns, err := client.CoreV1().Namespaces().Get(ctx, namespace, metav1.GetOptions{}) + if err != nil { + if k8serrors.IsNotFound(err) { + // Nothing to reclaim; the reconciler creates and labels it when the + // Gateway next provisions. Never create it here. + return false, nil + } + return false, fmt.Errorf("get namespace %s: %w", namespace, err) + } + if !hasManagementLabels(ns) { + // Not a namespace this control plane created. Refuse to claim it even + // though a Gateway records its name, so a mis-recorded namespace field can + // never adopt an unrelated namespace. + log.Printf("INFO namespace %s lacks HyperShell management labels; not backfilling instance label", namespace) + return false, nil + } + if existing := ns.Labels[InstanceLabel]; existing != "" { + if existing != instance { + log.Printf("INFO namespace %s is managed by instance %q, not %q; not backfilling instance label", namespace, existing, instance) + } + return false, nil + } + patch := fmt.Appendf(nil, `{"metadata":{"labels":{%q:%q}}}`, InstanceLabel, instance) + if _, err := client.CoreV1().Namespaces().Patch(ctx, namespace, types.MergePatchType, patch, metav1.PatchOptions{}); err != nil { + return false, fmt.Errorf("backfill instance label on namespace %s: %w", namespace, err) + } + log.Printf("INFO backfilled instance label %s=%s onto legacy namespace %s", InstanceLabel, instance, namespace) + return true, nil +} + // MarkGCEligible stamps the gc-eligible-since annotation with the given time if // it is not already present, returning the effective eligible-since time. The // timestamp is persisted on the namespace so the grace period survives diff --git a/components/control-plane/internal/gateway/namespace_test.go b/components/control-plane/internal/gateway/namespace_test.go index 29fd956f..3afb9bb0 100644 --- a/components/control-plane/internal/gateway/namespace_test.go +++ b/components/control-plane/internal/gateway/namespace_test.go @@ -265,6 +265,100 @@ func TestEnsureManagedNamespace(t *testing.T) { }) } +func TestBackfillInstanceLabel(t *testing.T) { + ctx := context.Background() + + t.Run("stamps instance on legacy managed namespace", func(t *testing.T) { + client := fake.NewSimpleClientset(managedNamespaceForInstance("openshell-gw", "", nil)) + labeled, err := BackfillInstanceLabel(ctx, client, "openshell-gw", "hypershell") + if err != nil { + t.Fatalf("BackfillInstanceLabel() error = %v", err) + } + if !labeled { + t.Errorf("labeled = false, want true for legacy namespace") + } + got, err := client.CoreV1().Namespaces().Get(ctx, "openshell-gw", metav1.GetOptions{}) + if err != nil { + t.Fatalf("get namespace: %v", err) + } + if got.Labels[InstanceLabel] != "hypershell" { + t.Errorf("instance label = %q, want hypershell", got.Labels[InstanceLabel]) + } + if got.Labels[ManagedLabel] != ManagedLabelValue || got.Labels[ManagedByLabel] != ManagedByValue { + t.Errorf("management labels lost during patch: %v", got.Labels) + } + }) + + t.Run("is a no-op when already labeled for this instance", func(t *testing.T) { + client := fake.NewSimpleClientset(managedNamespaceForInstance("openshell-gw", "hypershell", nil)) + labeled, err := BackfillInstanceLabel(ctx, client, "openshell-gw", "hypershell") + if err != nil { + t.Fatalf("BackfillInstanceLabel() error = %v", err) + } + if labeled { + t.Errorf("labeled = true, want false when label already present") + } + }) + + t.Run("never overwrites a foreign instance label", func(t *testing.T) { + client := fake.NewSimpleClientset(managedNamespaceForInstance("openshell-gw", "stage", nil)) + labeled, err := BackfillInstanceLabel(ctx, client, "openshell-gw", "hypershell") + if err != nil { + t.Fatalf("BackfillInstanceLabel() error = %v", err) + } + if labeled { + t.Errorf("labeled = true, want false for foreign instance") + } + got, err := client.CoreV1().Namespaces().Get(ctx, "openshell-gw", metav1.GetOptions{}) + if err != nil { + t.Fatalf("get namespace: %v", err) + } + if got.Labels[InstanceLabel] != "stage" { + t.Errorf("instance label overwritten to %q, want stage", got.Labels[InstanceLabel]) + } + }) + + t.Run("skips a namespace without management labels", func(t *testing.T) { + unmanaged := &corev1.Namespace{ObjectMeta: metav1.ObjectMeta{Name: "openshell-gw"}} + client := fake.NewSimpleClientset(unmanaged) + labeled, err := BackfillInstanceLabel(ctx, client, "openshell-gw", "hypershell") + if err != nil { + t.Fatalf("BackfillInstanceLabel() error = %v", err) + } + if labeled { + t.Errorf("labeled = true, want false for a namespace lacking management labels") + } + got, err := client.CoreV1().Namespaces().Get(ctx, "openshell-gw", metav1.GetOptions{}) + if err != nil { + t.Fatalf("get namespace: %v", err) + } + if _, ok := got.Labels[InstanceLabel]; ok { + t.Errorf("instance label stamped on an unmanaged namespace: %v", got.Labels) + } + }) + + t.Run("never creates an absent namespace", func(t *testing.T) { + client := fake.NewSimpleClientset() + labeled, err := BackfillInstanceLabel(ctx, client, "openshell-gone", "hypershell") + if err != nil { + t.Fatalf("BackfillInstanceLabel() error = %v", err) + } + if labeled { + t.Errorf("labeled = true, want false for an absent namespace") + } + if _, err := client.CoreV1().Namespaces().Get(ctx, "openshell-gone", metav1.GetOptions{}); !k8serrors.IsNotFound(err) { + t.Errorf("namespace was created by backfill, err = %v", err) + } + }) + + t.Run("refuses an empty instance identity", func(t *testing.T) { + client := fake.NewSimpleClientset(managedNamespaceForInstance("openshell-gw", "", nil)) + if _, err := BackfillInstanceLabel(ctx, client, "openshell-gw", ""); err == nil { + t.Fatalf("BackfillInstanceLabel() error = nil, want empty instance error") + } + }) +} + func TestMarkGCEligible(t *testing.T) { ctx := context.Background() now := time.Date(2026, 8, 17, 12, 0, 0, 0, time.UTC) diff --git a/components/control-plane/internal/reconciler/backfill.go b/components/control-plane/internal/reconciler/backfill.go new file mode 100644 index 00000000..e82fb1f3 --- /dev/null +++ b/components/control-plane/internal/reconciler/backfill.go @@ -0,0 +1,97 @@ +package reconciler + +import ( + "context" + "errors" + "fmt" + "log" + + pb "github.com/openshift-online/hypershell/components/api-server/pkg/api/grpc/hypershell/v1" + "github.com/openshift-online/hypershell/components/control-plane/internal/gateway" + "google.golang.org/grpc" + "k8s.io/client-go/kubernetes" +) + +// BackfillInstanceLabels stamps this control-plane instance's identity label onto +// the legacy gateway namespaces it already owns per its API server, so periodic +// GC -- which selects on hypershell.redhat.io/instance -- can reclaim them once +// their Gateway is deleted. It is a one-shot startup task. +// +// The normal reconcile path (EnsureManagedNamespace) migrates a live gateway's +// pre-label namespace only when the Gateway is actually reconciled. But a Gateway +// in a steady-state phase (Running/Provisioning/Degraded) is skipped by the +// reconciler's phase gate on (re)start, so its namespace would otherwise never +// gain the instance label and a later missed-delete would leak it past the +// instance-scoped sweep -- precisely the namespaces GC exists to reap. This +// backfill closes that gap by driving the migration from the Gateway inventory +// directly rather than through the gated reconcile. +// +// It is DB-driven and safe on shared clusters: it only ever touches namespaces a +// Gateway in THIS instance's API server records, and even then only namespaces +// that already carry both management labels and no foreign instance label (see +// gateway.BackfillInstanceLabel). Namespaces this instance does not know about -- +// including those another HyperShell owns -- are never read or written. It is +// idempotent and best-effort: a per-namespace failure is collected and the sweep +// continues, so one unreachable namespace cannot block the rest. It returns the +// number of namespaces newly labeled and a joined error of any per-namespace +// failures. +// +// Namespaces already orphaned (no Gateway row) before this instance ever labeled +// them cannot be reclaimed here: with no Gateway recording their name and only the +// two generic management labels, they are indistinguishable from another +// HyperShell's namespaces on a shared cluster, so claiming them is unsafe. Those +// require manual cleanup. +func BackfillInstanceLabels(ctx context.Context, client kubernetes.Interface, gwClient pb.GatewayServiceClient, instance string) (int, error) { + if instance == "" { + return 0, fmt.Errorf("refusing to backfill instance labels without a control-plane instance identity") + } + gateways, err := listAllGateways(ctx, gwClient) + if err != nil { + // A partial inventory would silently skip namespaces that need the label, + // leaving them to leak; fail so the caller can log and retry on the next + // restart rather than accept an incomplete backfill. + return 0, fmt.Errorf("list gateways for instance-label backfill: %w", err) + } + + var errs error + labeled := 0 + seen := make(map[string]struct{}, len(gateways)) + for _, gw := range gateways { + ns := gw.GetNamespace() + if ns == "" { + // A gateway with no recorded namespace gives nothing deterministic to + // label; the reconciler assigns and labels it when the Gateway + // provisions. Never synthesize a namespace name here. + continue + } + if _, dup := seen[ns]; dup { + continue + } + seen[ns] = struct{}{} + + did, err := gateway.BackfillInstanceLabel(ctx, client, ns, instance) + if err != nil { + errs = errors.Join(errs, fmt.Errorf("namespace %s: %w", ns, err)) + continue + } + if did { + labeled++ + } + } + return labeled, errs +} + +// RunInstanceLabelBackfill performs the startup instance-label backfill against a +// live gRPC connection and logs a summary. It is best-effort: a failure is logged +// but never aborts controller startup, because the periodic GC sweep still +// functions for already-labeled namespaces and the backfill is retried on the +// next restart. +func RunInstanceLabelBackfill(ctx context.Context, client kubernetes.Interface, conn *grpc.ClientConn, instance string) { + gwClient := pb.NewGatewayServiceClient(conn) + labeled, err := BackfillInstanceLabels(ctx, client, gwClient, instance) + if err != nil { + log.Printf("WARN instance-label backfill completed with errors (labeled=%d): %v", labeled, err) + return + } + log.Printf("INFO instance-label backfill complete: labeled %d legacy namespace(s) for instance %s", labeled, instance) +} diff --git a/components/control-plane/internal/reconciler/backfill_test.go b/components/control-plane/internal/reconciler/backfill_test.go new file mode 100644 index 00000000..64affcaf --- /dev/null +++ b/components/control-plane/internal/reconciler/backfill_test.go @@ -0,0 +1,159 @@ +package reconciler + +import ( + "context" + "fmt" + "testing" + + pb "github.com/openshift-online/hypershell/components/api-server/pkg/api/grpc/hypershell/v1" + "github.com/openshift-online/hypershell/components/control-plane/internal/gateway" + "google.golang.org/grpc" + corev1 "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/runtime" + "k8s.io/client-go/kubernetes/fake" + ktesting "k8s.io/client-go/testing" +) + +func legacyGatewayNamespace(name string) *corev1.Namespace { + return &corev1.Namespace{ + ObjectMeta: metav1.ObjectMeta{ + Name: name, + Labels: map[string]string{ + gateway.ManagedByLabel: gateway.ManagedByValue, + gateway.ManagedLabel: gateway.ManagedLabelValue, + }, + }, + } +} + +func foreignGatewayNamespace(name, instance string) *corev1.Namespace { + ns := legacyGatewayNamespace(name) + ns.Labels[gateway.InstanceLabel] = instance + return ns +} + +func gatewayWithNamespace(id, namespace string) *pb.Gateway { + return &pb.Gateway{ + Metadata: &pb.ObjectReference{Id: id}, + Namespace: namespace, + } +} + +// singlePageGatewayClient returns a fake that serves the whole inventory on the +// first page. listAllGateways stops once a short page is returned, so a slice +// smaller than the page size terminates paging after page 1. +func singlePageGatewayClient(gws []*pb.Gateway) *fakeGatewayClient { + return &fakeGatewayClient{ + listFn: func(ctx context.Context, in *pb.ListGatewaysRequest, opts ...grpc.CallOption) (*pb.ListGatewaysResponse, error) { + meta := &pb.ListMeta{Page: in.Page, Size: in.Size, Total: int32(len(gws))} + if in.Page > 1 { + return &pb.ListGatewaysResponse{Metadata: meta}, nil + } + return &pb.ListGatewaysResponse{Items: gws, Metadata: meta}, nil + }, + } +} + +func TestBackfillInstanceLabels(t *testing.T) { + ctx := context.Background() + + t.Run("labels legacy namespaces of live gateways and spares foreign ones", func(t *testing.T) { + client := fake.NewSimpleClientset( + legacyGatewayNamespace("openshell-a"), + legacyGatewayNamespace("openshell-b"), + foreignGatewayNamespace("openshell-c", "stage"), + ) + gwClient := singlePageGatewayClient([]*pb.Gateway{ + gatewayWithNamespace("id-a", "openshell-a"), + gatewayWithNamespace("id-b", "openshell-b"), + gatewayWithNamespace("id-c", "openshell-c"), + gatewayWithNamespace("id-missing", "openshell-not-present"), + gatewayWithNamespace("id-empty", ""), + }) + + labeled, err := BackfillInstanceLabels(ctx, client, gwClient, "hypershell") + if err != nil { + t.Fatalf("BackfillInstanceLabels() error = %v", err) + } + if labeled != 2 { + t.Errorf("labeled = %d, want 2", labeled) + } + for _, name := range []string{"openshell-a", "openshell-b"} { + got, err := client.CoreV1().Namespaces().Get(ctx, name, metav1.GetOptions{}) + if err != nil { + t.Fatalf("get namespace %s: %v", name, err) + } + if got.Labels[gateway.InstanceLabel] != "hypershell" { + t.Errorf("%s instance label = %q, want hypershell", name, got.Labels[gateway.InstanceLabel]) + } + } + foreign, err := client.CoreV1().Namespaces().Get(ctx, "openshell-c", metav1.GetOptions{}) + if err != nil { + t.Fatalf("get namespace openshell-c: %v", err) + } + if foreign.Labels[gateway.InstanceLabel] != "stage" { + t.Errorf("foreign namespace relabeled to %q, want stage", foreign.Labels[gateway.InstanceLabel]) + } + }) + + t.Run("aborts when the gateway list fails", func(t *testing.T) { + client := fake.NewSimpleClientset(legacyGatewayNamespace("openshell-a")) + gwClient := &fakeGatewayClient{ + listFn: func(ctx context.Context, in *pb.ListGatewaysRequest, opts ...grpc.CallOption) (*pb.ListGatewaysResponse, error) { + return nil, fmt.Errorf("boom") + }, + } + if _, err := BackfillInstanceLabels(ctx, client, gwClient, "hypershell"); err == nil { + t.Fatalf("BackfillInstanceLabels() error = nil, want list failure") + } + got, err := client.CoreV1().Namespaces().Get(ctx, "openshell-a", metav1.GetOptions{}) + if err != nil { + t.Fatalf("get namespace: %v", err) + } + if _, ok := got.Labels[gateway.InstanceLabel]; ok { + t.Errorf("namespace labeled despite list failure: %v", got.Labels) + } + }) + + t.Run("collects a per-namespace failure and continues", func(t *testing.T) { + client := fake.NewSimpleClientset( + legacyGatewayNamespace("openshell-a"), + legacyGatewayNamespace("openshell-b"), + ) + // Fail the Get for openshell-a only; openshell-b must still be labeled. + client.PrependReactor("get", "namespaces", func(action ktesting.Action) (bool, runtime.Object, error) { + if ga, ok := action.(ktesting.GetAction); ok && ga.GetName() == "openshell-a" { + return true, nil, fmt.Errorf("transient get error") + } + return false, nil, nil + }) + gwClient := singlePageGatewayClient([]*pb.Gateway{ + gatewayWithNamespace("id-a", "openshell-a"), + gatewayWithNamespace("id-b", "openshell-b"), + }) + + labeled, err := BackfillInstanceLabels(ctx, client, gwClient, "hypershell") + if err == nil { + t.Fatalf("BackfillInstanceLabels() error = nil, want a collected per-namespace error") + } + if labeled != 1 { + t.Errorf("labeled = %d, want 1 (openshell-b still labeled)", labeled) + } + got, err := client.CoreV1().Namespaces().Get(ctx, "openshell-b", metav1.GetOptions{}) + if err != nil { + t.Fatalf("get namespace openshell-b: %v", err) + } + if got.Labels[gateway.InstanceLabel] != "hypershell" { + t.Errorf("openshell-b instance label = %q, want hypershell", got.Labels[gateway.InstanceLabel]) + } + }) + + t.Run("refuses an empty instance identity", func(t *testing.T) { + client := fake.NewSimpleClientset() + gwClient := singlePageGatewayClient(nil) + if _, err := BackfillInstanceLabels(ctx, client, gwClient, ""); err == nil { + t.Fatalf("BackfillInstanceLabels() error = nil, want empty instance error") + } + }) +} diff --git a/components/control-plane/internal/reconciler/namespace.go b/components/control-plane/internal/reconciler/namespace.go index eeac0eac..9db22e81 100644 --- a/components/control-plane/internal/reconciler/namespace.go +++ b/components/control-plane/internal/reconciler/namespace.go @@ -45,11 +45,13 @@ const ( // this control-plane instance created but that no longer have a live Gateway in // this instance's API server. Other HyperShell instances on the same cluster are // ignored: the sweep selects on hypershell.redhat.io/instance=. -// Legacy gateway namespaces that predate the instance label must be labeled -// manually (both management labels, no instance label) before they become -// visible to a sweep; this reconciler never claims them itself. This reaps -// namespaces orphaned by a delete event missed while the control plane was -// down, and namespaces whose gateway failed to bootstrap and was then deleted. +// Legacy gateway namespaces that predate the instance label (both management +// labels, no instance label) are not visible to a sweep on their own; the +// one-shot startup backfill (BackfillInstanceLabels) stamps the instance label +// onto those this instance still owns per its API server, and this reconciler +// never claims an unlabeled namespace itself. This reaps namespaces orphaned by a +// delete event missed while the control plane was down, and namespaces whose +// gateway failed to bootstrap and was then deleted. // Reaping is best-effort and idempotent, and is delayed by a grace period // recorded on the namespace itself so it survives restarts. // diff --git a/components/control-plane/internal/supervisor/supervisor.go b/components/control-plane/internal/supervisor/supervisor.go new file mode 100644 index 00000000..d8e42ca9 --- /dev/null +++ b/components/control-plane/internal/supervisor/supervisor.go @@ -0,0 +1,44 @@ +package supervisor + +import ( + "context" + "log" + "time" +) + +// RestartCap bounds the backoff between restarts of a supervised background component. +const RestartCap = 30 * time.Second + +// Run runs fn in a loop so a failure in one background component +// cannot take down the others by exiting the whole process. Every fn here is +// expected to block until ctx is done and return ctx.Err() at that point; +// a return before then is treated as a failure of that component alone; it is retried +// with a capped exponential backoff. +func Run(ctx context.Context, name string, fn func(context.Context) error) { + RunWithBackoff(ctx, name, time.Second, fn) +} + +// RunWithBackoff runs fn in a loop with a customizable initial backoff. +func RunWithBackoff(ctx context.Context, name string, initialBackoff time.Duration, fn func(context.Context) error) { + backoff := initialBackoff + for { + err := fn(ctx) + if ctx.Err() != nil { + return + } + if err == nil { + log.Printf("WARN %s exited unexpectedly with no error, restarting in %s", name, backoff) + } else { + log.Printf("WARN %s exited with error, restarting in %s: %v", name, backoff, err) + } + select { + case <-ctx.Done(): + return + case <-time.After(backoff): + } + backoff *= 2 + if backoff > RestartCap { + backoff = RestartCap + } + } +} diff --git a/components/control-plane/internal/supervisor/supervisor_test.go b/components/control-plane/internal/supervisor/supervisor_test.go new file mode 100644 index 00000000..013013cf --- /dev/null +++ b/components/control-plane/internal/supervisor/supervisor_test.go @@ -0,0 +1,33 @@ +package supervisor + +import ( + "context" + "errors" + "testing" + "time" +) + +func TestRun(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + calls := 0 + fn := func(c context.Context) error { + calls++ + if calls == 1 { + return nil // Return nil on first call, should restart + } + if calls == 2 { + return errors.New("some error") // Return error on second call, should restart + } + // On third call, cancel context to break the loop + cancel() + return c.Err() + } + + RunWithBackoff(ctx, "test-component", time.Millisecond, fn) + + if calls != 3 { + t.Errorf("expected fn to be called 3 times, got %d", calls) + } +} diff --git a/deploy/base/controller.yaml b/deploy/base/controller.yaml index ca80ebc6..0b356a54 100644 --- a/deploy/base/controller.yaml +++ b/deploy/base/controller.yaml @@ -55,6 +55,10 @@ spec: ports: - containerPort: 9443 name: provisioner + # These probes are coupled to the service account provisioner port (9443). + # If HYPERSHELL_SERVICE_ACCOUNT_PROVISIONER_BIND_ADDRESS is ever empty, + # the provisioner is disabled, nothing binds port 9443, and the controller will fail probes. + # Overlays that disable the provisioner must also override or remove these probes. startupProbe: tcpSocket: port: provisioner diff --git a/specs/platform/openshell-gateway-namespace-gc.spec.md b/specs/platform/openshell-gateway-namespace-gc.spec.md index 54d2ca93..950de8c2 100644 --- a/specs/platform/openshell-gateway-namespace-gc.spec.md +++ b/specs/platform/openshell-gateway-namespace-gc.spec.md @@ -57,9 +57,11 @@ Namespace creation and provisioning mechanics are defined in namespaces (`openshell-db-`). A namespace owned by a different instance is never listed, annotated, or reaped. A gateway namespace that carries both management labels but no instance label is unlabeled leftover. Periodic GC - SHALL leave it unlabeled. An operator labels it for the owning instance when - they want that instance's sweep to consider it. ManagedDatabase namespaces and - namespaces already labeled for another instance are never claimed. + SHALL leave it unlabeled. A namespace still recorded by a live Gateway in this + instance's API server is claimed for this instance by the startup backfill (see + "Backfill Instance Labels on Startup") and by live reconcile; an unlabeled + leftover with no such Gateway is claimed only by an operator. ManagedDatabase + namespaces and namespaces already labeled for another instance are never claimed. A Gateway pointed at a pre-existing or shared namespace can never cause that namespace to be reaped. - **Orphaned namespace** - a gateway namespace (matching the gateway prefix, not @@ -280,6 +282,57 @@ controllers on the same cluster SHALL NOT share an instance identity. - THEN it SHALL NOT overwrite the instance label - AND it SHALL NOT deploy into or delete that namespace +### Requirement: Backfill Instance Labels on Startup + +On startup, before the periodic garbage collector's first sweep, the control +plane SHALL stamp `hypershell.redhat.io/instance` onto the legacy gateway +namespaces it still owns per its API server, so periodic GC can reclaim them once +their Gateway is deleted. The backfill is required because the reconciler's phase +gate skips reconciliation of a Gateway already in a steady-state phase +(`Running`/`Provisioning`/`Degraded`) on (re)start, so `EnsureManagedNamespace` +never runs for it and its pre-label namespace would otherwise never gain the +instance label; a later missed-delete would then leak that namespace past the +instance-scoped sweep. + +The backfill SHALL be driven from the Gateway inventory in this instance's API +server: for each Gateway that records a namespace, it stamps the instance label +onto that namespace only if the namespace already carries both management labels +and no instance label. It SHALL NOT create a namespace, SHALL NOT touch a +namespace lacking the management labels, and SHALL NOT overwrite an instance +label that already identifies a different instance. It SHALL be idempotent and +best-effort: a per-namespace failure is recorded and the backfill continues, and +a failure of the whole backfill SHALL NOT block controller startup (the periodic +sweep still functions for already-labeled namespaces and the backfill is retried +on the next restart). + +A namespace already orphaned before this instance ever labeled it (no Gateway +records its name, only the two management labels) cannot be reclaimed by the +backfill: on a shared cluster it is indistinguishable from a namespace another +HyperShell owns, so claiming it is unsafe. Such namespaces require manual +cleanup. + +#### Scenario: Startup backfill labels a live gateway's legacy namespace + +- GIVEN a namespace that carries the two management labels but no instance label +- AND a live Gateway in this instance's API server records that namespace +- WHEN the control plane starts and runs the instance-label backfill +- THEN it SHALL stamp `hypershell.redhat.io/instance` for this instance onto that + namespace + +#### Scenario: Startup backfill does not claim a foreign or unmanaged namespace + +- GIVEN a live Gateway in this instance's API server records a namespace +- AND that namespace either carries `hypershell.redhat.io/instance` for a + different instance, or lacks the management labels +- WHEN the control plane runs the instance-label backfill +- THEN it SHALL NOT stamp or overwrite the instance label on that namespace + +#### Scenario: Startup backfill never creates a namespace + +- GIVEN a live Gateway that records a namespace which does not exist on the cluster +- WHEN the control plane runs the instance-label backfill +- THEN it SHALL NOT create that namespace + ### Requirement: Grace Period Prevents Premature Deletion The control plane SHALL NOT delete an orphaned namespace immediately. It SHALL @@ -417,7 +470,8 @@ real-time guarantee. | Require this instance's identity label before periodic GC | Two HyperShell controllers on one cluster share the generic management labels. Without a value unique to that controller (`hypershell.redhat.io/instance=`), instance B's sweep treats instance A's live gateways as orphans (they are absent from B's API server) and would delete them after the grace period. The identity is the controller pod's namespace from the downward API so it cannot be a copied static string. | | Periodic GC never claims unlabeled legacy gateway namespaces | An earlier design had each sweep stamp the instance label onto unlabeled leftovers before evaluating them for orphaning, so a missed-delete orphan predating the instance label would not leak forever. In practice this reclaimed the label on every sweep for a namespace whose instance label never durably stuck (e.g. a naming or list-vs-read inconsistency), which reset `gc-eligible-since` to "now" on every tick and made the namespace look freshly orphaned forever, so it was never reaped. Claiming unlabeled leftovers is now an explicit operator action instead. | | Require BOTH management labels plus this instance's identity before deleting | Defense in depth: even if a label selector over-returns, a namespace not created by this control-plane instance (another HyperShell, a shared namespace, or a pre-existing namespace) is never deleted by periodic GC. | -| Stamp the instance label on create and on live reconcile only | Periodic GC can only reap namespaces this instance labeled. Stamping at create covers new gateways; stamping on live reconcile migrates this instance's still-live pre-label namespaces. Unlabeled leftovers with no live Gateway stay unlabeled and excluded from GC until an operator labels them. | +| Stamp the instance label on create, on live reconcile, and via a startup backfill | Periodic GC can only reap namespaces this instance labeled. Stamping at create covers new gateways; stamping on live reconcile migrates pre-label namespaces as gateways reconcile. But the reconciler's phase gate skips a Gateway already in a steady-state phase on (re)start, so live reconcile alone never migrates a Running gateway's legacy namespace. A one-shot startup backfill, driven from the Gateway inventory, closes that gap before the first sweep. Unlabeled leftovers with no live Gateway stay unlabeled and excluded from GC until an operator labels them. | +| Startup backfill is DB-driven and cannot reclaim already-orphaned unlabeled namespaces | The backfill only ever labels namespaces a Gateway in this instance's API server records, so on a shared cluster it never touches a namespace another HyperShell owns. A namespace already orphaned before the instance label existed has no Gateway recording it and only the two generic management labels, making it indistinguishable from another HyperShell's namespace; claiming it is unsafe, so it is left for manual cleanup. An earlier attempt to reclaim such namespaces from cluster state alone was abandoned as too error-prone for exactly this reason. | | Grace period persisted on the namespace annotation | The delay must survive control-plane restarts; storing `gc-eligible-since` on the namespace makes the timer durable without a separate store. | | Abort the whole sweep if Gateways cannot be listed | An empty or failed Gateway list would make every managed namespace look orphaned; aborting is the only safe response to avoid mass reaping of live namespaces. | | Delete is best-effort and not gated on sandbox count | Deletion is idempotent - process the delete, remove the namespace, and if it is already gone consider the delete done. The sandbox count is a warning surfaced to the operator, not a backend precondition. |