feat(decofile): add s3 target to serve decofile over HTTP (off etcd)

ARCHITECTURE.md

@@ -242,6 +242,101 @@ sequenceDiagram
     end
 ```
 
+## Decofile Delivery: S3 Target (etcd Offload)
+
+Content-heavy sites (many/large blocks — e.g. a blog with full-HTML posts inline)
+can push the merged decofile's brotli+base64 size past the ~1MB etcd ConfigMap
+ceiling, at which point the Kubernetes API server rejects the ConfigMap write and
+the site can no longer publish. `spec.target: "s3"` is an **opt-in, per-site**
+alternative: the operator uploads the decofile to S3 as plain JSON and points
+pods at it over HTTP instead of mounting a ConfigMap. Fully reversible — flip
+`target` back to `configmap` (the default) and the next reconcile reverts.
+
+### Flow 7a: Cold Start (target=s3)
+
+```mermaid
+sequenceDiagram
+    participant User
+    participant K8s as Kubernetes API
+    participant Ctrl as Decofile Controller
+    participant S3 as S3 Bucket
+    participant Webhook as Mutating Webhook
+    participant Pod as Site Pod
+
+    User->>K8s: Create/Update Decofile (spec.target: s3)
+    K8s->>Ctrl: Reconcile Event
+    Ctrl->>Ctrl: Retrieve source (github/inline) → JSON
+    Ctrl->>Ctrl: SHA-256 hash content
+    alt Hash unchanged (& commit unchanged for github)
+        Ctrl->>Ctrl: Skip upload — nothing to do
+    else Hash changed
+        Ctrl->>S3: PutObject (raw JSON, no compression)
+        Ctrl->>K8s: Update Status (ContentHash, S3URL)
+    end
+
+    Note over User,Pod: Separately — Service creation/update
+    User->>K8s: Create Knative Service (deco.sites/decofile-inject)
+    K8s->>Webhook: Intercept CREATE/UPDATE
+    Webhook->>K8s: Get Decofile (target=s3)
+    Webhook->>Webhook: Derive S3 key + URL (same formula as controller —<br/>Decofile.S3ObjectKey(), no dependency on reconcile order)
+    Webhook->>Webhook: Set env DECO_RELEASE=https://host/key<br/>(NO volume/volumeMount injected)
+    Webhook->>Webhook: Merge host into DECO_ALLOWED_AUTHORITIES<br/>(preserves runtime defaults if unset)
+    K8s->>Pod: Create Pod with injected env
+    Pod->>S3: GET https://host/key (plain HTTPS, no AWS auth)
+    S3->>Pod: 200 OK — decofile JSON
+    Pod->>Pod: fetch().then(JSON.parse) — no brotli/base64 decode
+```
+
+### Flow 7b: Hot Reload (target=s3)
+
+Identical to **Flow 4** (Change Notification) — the s3 target reuses
+`NotifyPodsForDecofile` unchanged. The only difference from the ConfigMap path
+is *what* changed (an S3 object, not a ConfigMap) and that the reconciler
+pushes the **uncompressed** JSON in the notify payload (same as it always has —
+`NotifyPodsForDecofile` was never brotli-aware; compression is a ConfigMap-side
+concern only). Pods that implement `/.decofile/reload` don't need to know which
+target delivered their *next* update; only the *initial* fetch path (mounted
+file vs. HTTP GET) differs.
+
+### s3 vs configmap: what changes
+
+| | `configmap` (default) | `s3` |
+|---|---|---|
+| Storage | ConfigMap key `decofile.bin` (brotli+base64) | S3 object (plain JSON) |
+| Size ceiling | ~1MB (etcd) | none (S3 object limit is 5TB) |
+| Cold start | Volume mount, `DECO_RELEASE=file://…` | HTTP env only, `DECO_RELEASE=https://…` |
+| Change detection | ConfigMap data diff | SHA-256 content hash (`Status.ContentHash`) |
+| Hot reload | `NotifyPodsForDecofile` (unchanged) | `NotifyPodsForDecofile` (unchanged) |
+| GC | Owned by Decofile (owner ref) — cascades | **Not GC'd** — a deleted Decofile leaves its S3 object; rely on a bucket lifecycle rule |
+| `DECO_ALLOWED_AUTHORITIES` | untouched | webhook merges the S3/CDN host in, preserving runtime defaults |
+
+### Design decisions
+
+- **No compression.** The runtime's HTTP decofile provider
+  (`deco/engine/decofile/fetcher.ts`) does `fetch() → JSON.parse()` with no
+  brotli/base64 decode — that path only exists for `file://….bin`. Serving
+  compressed bytes would require a runtime change; S3 has no size pressure
+  forcing that trade, so the s3 target serves raw JSON. (`gzip` +
+  `Content-Encoding` — which `fetch` auto-decodes — is a possible future
+  optimization, not required.)
+- **Content-hash gate, not ConfigMap-diff.** There's no "get the existing
+  object and compare" step for S3 (no cheap read-before-write like the
+  ConfigMap `Get`); a SHA-256 of the retrieved JSON, stored in
+  `Status.ContentHash`, decides whether to re-upload and re-notify. For a
+  `github` source this is checked *after* the (cheaper) commit-unchanged
+  short-circuit already used by the configmap path.
+- **Deterministic key derivation shared by both sides.** `Decofile.S3ObjectKey()`
+  is called by both the controller (upload) and the webhook (URL for
+  `DECO_RELEASE`) so the webhook never needs to wait for a reconcile to know
+  where the object will be — same reasoning as `ConfigMapName()` for the
+  existing target.
+- **`DECO_ALLOWED_AUTHORITIES` is additive, not overwritten.** Setting that env
+  var **replaces** the runtime's built-in default list (`configs.decocdn.com`,
+  `configs.deco.cx`, `admin.deco.cx`, `localhost`) rather than appending to it
+  — so the webhook always merges the s3 host into whatever's already on the
+  container (existing value or the defaults), never blindly setting it to just
+  the new host.
+
 ## Key Design Decisions
 
 ### 1. Compression Strategy

api/v1alpha1/decofile_types.go

@@ -17,6 +17,8 @@ limitations under the License.
 package v1alpha1
 
 import (
+	"strings"
+
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	runtime "k8s.io/apimachinery/pkg/runtime"
 )
@@ -37,6 +39,11 @@ const (
 	TargetConfigMap = "configmap"
 	// TargetTanstackKV runs a Job that pushes the decofile to Cloudflare KV.
 	TargetTanstackKV = "tanstack-kv"
+	// TargetS3 uploads the merged decofile JSON to S3 and points the runtime at
+	// it over HTTP (DECO_RELEASE=https://…) instead of mounting a ConfigMap.
+	// Escapes the ~1MB etcd ConfigMap ceiling for content-heavy sites. Bucket /
+	// region / public host come from operator env (DECOFILE_S3_*).
+	TargetS3 = "s3"
 )
 
 // DecofileSpec defines the desired state of Decofile.
@@ -65,7 +72,9 @@ type DecofileSpec struct {
 	// "configmap" (default) writes a ConfigMap and notifies Knative pods.
 	// "tanstack-kv" runs a self-cleaning Job that pushes the decofile to Cloudflare
 	// KV — the fast-deploy content path for TanStack/Workers sites.
-	// +kubebuilder:validation:Enum=configmap;tanstack-kv
+	// "s3" uploads the decofile to S3 and serves it over HTTP (no ConfigMap) —
+	// the path for content-heavy sites that would exceed the etcd ConfigMap limit.
+	// +kubebuilder:validation:Enum=configmap;tanstack-kv;s3
 	// +kubebuilder:default=configmap
 	// +optional
 	Target string `json:"target,omitempty"`
@@ -146,6 +155,15 @@ type DecofileStatus struct {
 	// JobName is the K8s Job name for the current tanstack-kv sync (target=tanstack-kv).
 	// +optional
 	JobName string `json:"jobName,omitempty"`
+
+	// ContentHash is the SHA-256 of the last delivered decofile JSON. Used by the
+	// s3 target to skip re-upload/notify when content is unchanged.
+	// +optional
+	ContentHash string `json:"contentHash,omitempty"`
+
+	// S3URL is the HTTP URL the runtime reads from when target=s3.
+	// +optional
+	S3URL string `json:"s3URL,omitempty"`
 }
 
 // +kubebuilder:object:root=true
@@ -165,6 +183,26 @@ func (d *Decofile) ConfigMapName() string {
 	return "decofile-" + d.Name
 }
 
+// DeploymentIdOrName returns spec.deploymentId, defaulting to the object name.
+func (d *Decofile) DeploymentIdOrName() string {
+	if d.Spec.DeploymentId != "" {
+		return d.Spec.DeploymentId
+	}
+	return d.Name
+}
+
+// S3ObjectKey returns the deterministic S3 key for this Decofile's config,
+// derived the same way by the reconciler (upload) and the Service webhook
+// (DECO_RELEASE URL) so neither depends on status. prefix is the operator-wide
+// DECOFILE_S3_PREFIX (may be empty).
+func (d *Decofile) S3ObjectKey(prefix string) string {
+	key := d.Namespace + "/" + d.DeploymentIdOrName() + "/decofile.json"
+	if prefix != "" {
+		return strings.Trim(prefix, "/") + "/" + key
+	}
+	return key
+}
+
 // +kubebuilder:object:root=true
 
 // DecofileList contains a list of Decofile.

api/v1alpha1/decofile_types_test.go

@@ -0,0 +1,41 @@
+/*
+Copyright 2025.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+*/
+
+package v1alpha1
+
+import (
+	"testing"
+
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+)
+
+// The S3 object key must be derived identically by the reconciler (upload) and
+// the Service webhook (DECO_RELEASE URL), so this pins the shared shape.
+func TestDecofileS3ObjectKey(t *testing.T) {
+	df := &Decofile{
+		ObjectMeta: metav1.ObjectMeta{Name: "dep-123", Namespace: "sites-econverse"},
+	}
+	cases := []struct {
+		name         string
+		deploymentId string
+		prefix       string
+		want         string
+	}{
+		{"name as deploymentId, no prefix", "", "", "sites-econverse/dep-123/decofile.json"},
+		{"explicit deploymentId", "abc", "", "sites-econverse/abc/decofile.json"},
+		{"prefix trimmed", "", "decofiles/", "decofiles/sites-econverse/dep-123/decofile.json"},
+		{"prefix without slash", "", "decofiles", "decofiles/sites-econverse/dep-123/decofile.json"},
+	}
+	for _, tc := range cases {
+		t.Run(tc.name, func(t *testing.T) {
+			df.Spec.DeploymentId = tc.deploymentId
+			if got := df.S3ObjectKey(tc.prefix); got != tc.want {
+				t.Fatalf("S3ObjectKey(%q) = %q, want %q", tc.prefix, got, tc.want)
+			}
+		})
+	}
+}

chart/templates/customresourcedefinition-decofiles.deco.sites.yaml

@@ -114,9 +114,12 @@ spec:
                   "configmap" (default) writes a ConfigMap and notifies Knative pods.
                   "tanstack-kv" runs a self-cleaning Job that pushes the decofile to Cloudflare
                   KV — the fast-deploy content path for TanStack/Workers sites.
+                  "s3" uploads the decofile to S3 and serves it over HTTP (no ConfigMap) —
+                  the path for content-heavy sites that would exceed the etcd ConfigMap limit.
                 enum:
                 - configmap
                 - tanstack-kv
+                - s3
                 type: string
             required:
             - source
@@ -191,6 +194,11 @@ spec:
                 description: ConfigMapName is the name of the ConfigMap created for
                   this Decofile
                 type: string
+              contentHash:
+                description: |-
+                  ContentHash is the SHA-256 of the last delivered decofile JSON. Used by the
+                  s3 target to skip re-upload/notify when content is unchanged.
+                type: string
               githubCommit:
                 description: GitHubCommit stores the commit SHA if using GitHub source
                 type: string
@@ -202,6 +210,9 @@ spec:
                 description: LastUpdated is the timestamp of the last update
                 format: date-time
                 type: string
+              s3URL:
+                description: S3URL is the HTTP URL the runtime reads from when target=s3.
+                type: string
               sourceType:
                 description: SourceType indicates which source was used (inline or
                   github)

chart/templates/deployment-operator-controller-manager.yaml

@@ -47,7 +47,7 @@ spec:
         command:
         - /manager
         image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
-        {{- if or (and .Values.github (or .Values.github.token .Values.github.existingSecret)) .Values.operatorApi.existingSecret (and .Values.valkey (get .Values.valkey "sentinelUrls")) .Values.cfworkers.existingSecret .Values.cfworkers.builderImage .Values.cfworkers.artifactsBucket .Values.s3.region .Values.s3.logsBucket .Values.s3.stateBucket .Values.build.serviceAccount .Values.build.roleArn .Values.build.nodeSelector .Values.build.tolerations (and .Values.fastDeploy .Values.fastDeploy.syncerImage) }}
+        {{- if or (and .Values.github (or .Values.github.token .Values.github.existingSecret)) .Values.operatorApi.existingSecret (and .Values.valkey (get .Values.valkey "sentinelUrls")) .Values.cfworkers.existingSecret .Values.cfworkers.builderImage .Values.cfworkers.artifactsBucket .Values.s3.region .Values.s3.logsBucket .Values.s3.stateBucket .Values.build.serviceAccount .Values.build.roleArn .Values.build.nodeSelector .Values.build.tolerations (and .Values.fastDeploy .Values.fastDeploy.syncerImage) (and .Values.decofileS3 .Values.decofileS3.bucket) }}
         env:
         {{- if and .Values.github .Values.github.existingSecret }}
         - name: GITHUB_TOKEN
@@ -135,6 +135,16 @@ spec:
         - name: DECOFILE_SYNCER_IMAGE
           value: {{ .Values.fastDeploy.syncerImage | quote }}
         {{- end }}
+        {{- if and .Values.decofileS3 .Values.decofileS3.bucket }}
+        - name: DECOFILE_S3_BUCKET
+          value: {{ .Values.decofileS3.bucket | quote }}
+        - name: DECOFILE_S3_REGION
+          value: {{ .Values.decofileS3.region | quote }}
+        - name: DECOFILE_S3_PUBLIC_HOST
+          value: {{ .Values.decofileS3.publicHost | quote }}
+        - name: DECOFILE_S3_PREFIX
+          value: {{ .Values.decofileS3.prefix | quote }}
+        {{- end }}
         {{- if .Values.operatorApi.existingSecret }}
         {{- if .Values.operatorApi.addr }}
         - name: OPERATOR_API_ADDR

chart/values.yaml

@@ -116,6 +116,19 @@ s3:
   logsBucket: ""        # S3 bucket for build logs
   stateBucket: ""       # S3 bucket for site state
 
+# Decofile S3 delivery (target=s3): serves the decofile to pods over HTTP from
+# S3 instead of a ConfigMap, escaping the ~1MB etcd ConfigMap limit. Inert
+# unless `bucket` is set. The controller writes to S3 with credentials from its
+# EKS Pod Identity association (same mechanism as the build buckets) — that
+# role needs s3:PutObject on the bucket. `publicHost` is the runtime-facing host
+# (a CloudFront/VPC-restricted domain fronting the private bucket — the decofile
+# can contain secrets, so the bucket must NOT be public-read).
+decofileS3:
+  bucket: ""        # S3 bucket name (required to enable)
+  region: ""        # bucket region (co-locate with the EKS cluster)
+  publicHost: ""    # host for DECO_RELEASE URL, e.g. configs.decocdn.com
+  prefix: ""        # optional key prefix, e.g. "decofiles"
+
 # Build job config — shared across all build platforms
 build:
   serviceAccount: ""    # K8s ServiceAccount for builder pods (IRSA)

cmd/main.go

@@ -357,11 +357,20 @@ func main() {
 			decositesv1alpha1.TargetTanstackKV,
 			deploy.NewTanstackKV(tkvCfg),
 		)
+		// s3 target: HTTP-delivered decofile for content-heavy sites. Inert
+		// (nil) unless DECOFILE_S3_BUCKET is set.
+		s3Uploader, s3err := controller.NewS3UploaderFromEnv(context.Background())
+		if s3err != nil {
+			setupLog.Error(s3err, "decofile s3 target disabled: invalid DECOFILE_S3_* config")
+		} else if s3Uploader != nil {
+			setupLog.Info("decofile s3 target enabled")
+		}
 		if err = (&controller.DecofileReconciler{
 			Client:     mgr.GetClient(),
 			Scheme:     mgr.GetScheme(),
 			HTTPClient: httpClient,
 			FastDeploy: fastDeployRegistry,
+			S3:         s3Uploader,
 		}).SetupWithManager(mgr); err != nil {
 			setupLog.Error(err, "unable to create controller", "controller", "Decofile")
 			os.Exit(1)

config/crd/bases/deco.sites_decofiles.yaml

@@ -115,9 +115,12 @@ spec:
                   "configmap" (default) writes a ConfigMap and notifies Knative pods.
                   "tanstack-kv" runs a self-cleaning Job that pushes the decofile to Cloudflare
                   KV — the fast-deploy content path for TanStack/Workers sites.
+                  "s3" uploads the decofile to S3 and serves it over HTTP (no ConfigMap) —
+                  the path for content-heavy sites that would exceed the etcd ConfigMap limit.
                 enum:
                 - configmap
                 - tanstack-kv
+                - s3
                 type: string
             required:
             - source
@@ -192,6 +195,11 @@ spec:
                 description: ConfigMapName is the name of the ConfigMap created for
                   this Decofile
                 type: string
+              contentHash:
+                description: |-
+                  ContentHash is the SHA-256 of the last delivered decofile JSON. Used by the
+                  s3 target to skip re-upload/notify when content is unchanged.
+                type: string
               githubCommit:
                 description: GitHubCommit stores the commit SHA if using GitHub source
                 type: string
@@ -203,6 +211,9 @@ spec:
                 description: LastUpdated is the timestamp of the last update
                 format: date-time
                 type: string
+              s3URL:
+                description: S3URL is the HTTP URL the runtime reads from when target=s3.
+                type: string
               sourceType:
                 description: SourceType indicates which source was used (inline or
                   github)