feat: load the HTTP catalog with validated cache fallback

[lloydrichards]

Build one loader that the CLI and Recipe Builder can use to fetch, validate, and cache a catalog. Return a validated catalog plus its source, content digest, freshness state, and any cache warning. Both clients must use the same rules for accepting current or cached data.

Parent: #275. Blocked by #249. This issue provides the JSON format, loader API, cache interface, and typed errors consumed by #272, #273, and #274.

Deliver the format and loader

• Define the document in @repo/domain with formatVersion: 1, catalogId, requiredCapabilities, targets, and modules.
• Export the full official definitions as JSON. Decode through Effect Schema so values such as TargetIdentity regain their methods.
• Freeze the v1 capability set for supported tokens and contribution operations. Reject exports that exceed it. Adding a definition must not increase requirements for existing v1 clients. Keep these interpreter capabilities separate from generated-project supportedRuntimes.
• Validate each source's structure before composition. Use refactor: compose and inject catalog definitions across scaffold consumers #249 to validate references and ownership after composition. The official milestone loads one source, but a future fragment must be able to reference another configured source.
• Compute a digest of the received UTF-8 document after HTTP content decoding. Use it for diagnostics and cache integrity, not publisher authentication or a project version pin.
• Keep one decoded catalog for the entire calling operation. A later load may return newer content.

Put the shared loader and runtime-neutral cache interface under packages/scaffold/src/service/catalog/. Export them through the browser-safe package entrypoint. Inject Effect HTTP and cache services. Use an in-memory test cache here; #273 owns filesystem persistence and #274 owns browser persistence.

Define cache behavior

Attempt a current fetch on every new load. Use ETag or Last-Modified when available. Key cache entries by normalized source URL. Store the document bytes, digest, HTTP validators, and last successful fetch or revalidation time.

Result	Required behavior
Valid HTTP 200 document	Validate the complete effective catalog, then replace the cache entry and return current data.
HTTP 304 with a usable entry	Return that document and update its successful revalidation time.
HTTP 304 without a usable entry	Retry once without validators. Fail if the server still supplies no usable document.
Transport failure, timeout, HTTP 408, 429, or 5xx	Return a compatible validated cache entry with a stale-data warning. Fail if no such entry exists.
Other failed HTTP status, invalid JSON, HTML response, invalid catalog, or unsupported protocol or capability	Return a typed error. Do not hide the failure behind an older cache entry.
Cache read failure or corrupt or incompatible entry	Attempt the network. Treat that cache entry as unavailable.
Cache write failure after a valid fetch	Return the valid document with a persistence warning. Do not fail the current operation.

Recheck cached byte integrity and engine compatibility before reuse. Age alone does not invalidate a compatible entry. Never use another URL's entry or replace valid cached data with an invalid response. Cache writes must publish a whole entry atomically; client adapters implement that requirement.

Do not disable TLS verification. Browser fetch errors may hide the underlying network or CORS cause, so do not invent a diagnosis. Return warning data that lets each client show the source and last validation time. #273 renders warnings to stderr; #274 renders a persistent notice.

Limit decompressed input to 8 MiB and the total load to 15 seconds, including the single 304 recovery request. Check the byte limit while receiving data. Cancellation must cancel the request and release streams. If measurements justify changing a limit, record the reason and retain boundary tests.

Done when

• The complete official catalog survives export and decode with its data and schema-class behavior intact.
• Unsupported formats, capabilities, and invalid composed catalogs produce distinct actionable errors before planning.
• Controlled HTTP tests cover every row of the cache table, byte limits, cancellation, and the total timeout.
• A fake clock makes warning timestamps deterministic.
• Changing the server document affects a later load but cannot replace an already returned catalog.
• The loader works through both host and browser-safe imports without host filesystem dependencies.

Keep out of this issue

Filesystem and IndexedDB adapters, UI messages, project configuration fields, hosted assets, deployment, community source selection, and changes to generated templates. #272 owns StackConfig.$schema and publication.

Validate

Run bun format, bun lint, bun run type-check, and scoped bun run test --filter=<workspace> commands. Never use bun test. Tests must use controlled HTTP and cache fixtures rather than the public registry. Run bun run okf:check for knowledge changes.

[pullfrog]

No duplicate found. #249 is the prerequisite for injected catalog composition; #272 publishes the wire output, while #273 and #274 own CLI and browser persistence and UI rather than this shared loader.

Suggested implementation sequence:

1. After refactor: compose and inject catalog definitions across scaffold consumers #249, add the v1 document and typed load errors in packages/domain/src/Catalog.ts; encode the full authoring definitions and decode with Effect Schema so nested TargetIdentity values retain their methods. Round trip the bundled catalog and freeze the supported capability baseline.
2. Add pure structural, duplicate ID, and post-composition graph validation in @repo/catalog. Reject unsupported capabilities and invalid definitions before constructing CatalogService; do not require future individual community fragments to have closed graphs.
3. Add a browser-safe loader under packages/scaffold/src/service/catalog/ with injected HTTP, clock, and cache services. Bound streamed decompressed bytes to 8 MiB and total load to 15 seconds; hash received bytes, keep a single immutable validated document per operation, and implement conditional refresh, valid-entry-only 304 reuse, one unconditional 304 recovery, and explicit fallback/error classification. Revalidate cache bytes and compatibility; never replace the last good entry on invalid 200, and warn rather than fail on persistence errors.
4. Test with controlled HTTP and in-memory cache fixtures, including mutable subsequent loads, 304 without a usable entry, invalid cached bytes, permanent versus transient errors, cancellation, and failed writes. Expose source URL, digest, validation time, and warning/error types for feat: load the official registry in CLI commands #273/feat: load the official registry in Recipe Builder #274 before their adapters begin; keep endpoint and deployment checks in feat: publish the official catalog and StackConfig JSON Schema #272/feat: deliver and qualify the official HTTP catalog registry #275.

Current boundary to watch: CatalogService still builds indexes from targetRegistry/moduleRegistry, and scaffold layers such as RecipePreviewService.layer provide their own CatalogService.layer. The loader must consume #249’s injectable composition boundary rather than bypassing it.

Task list (3/3 completed)

• Inspect issue and search for confirmed duplicates or useful related issues
• Investigate relevant catalog loading code and formulate actionable scope
• Post one concise issue response

  ^{｜ Implement a fix ➔ ｜ View workflow run ｜ via Pullfrog ｜ Using GPT Sol ｜ 𝕏}

[lloydrichards]

Implementation: #280, stacked on #278 (#249). The v1 JSON document, fixed interpreter capability set, official export, shared HTTP loader, and runtime-neutral cache interface are in review.

Local evidence: bun format, bun lint, bun run type-check, bun run test --filter=@repo/scaffold (142 scaffold tests), bun run okf:check, and the docs production build passed. Controlled HTTP tests cover the cache table, strict official export, byte limit and stream release, timeout cleanup, caller cancellation, fake-clock stale timestamp, and later-load updates.

Clean-checkout CI for #280 is pending. Filesystem and browser persistence, client wiring, hosted assets, and deployed HTTP behavior belong to #273, #274, and #272.