Skip to content

Add trustBundle as a SystemInfo volume data source - #941

Open
Max Thompson (thompsonmax) wants to merge 2 commits into
agent-substrate:mainfrom
thompsonmax:ctb-source
Open

Add trustBundle as a SystemInfo volume data source#941
Max Thompson (thompsonmax) wants to merge 2 commits into
agent-substrate:mainfrom
thompsonmax:ctb-source

Conversation

@thompsonmax

@thompsonmax Max Thompson (thompsonmax) commented Aug 13, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Part of #932 (PR 1 of 3). Adds the user-declarable trustBundle data source for SystemInfo volumes (#802) and the end-to-end proof that the projected anchors work against the MITM egress gateway. Live refresh for running actors (PR 2) and auto-injection (PR 3) come separately.

What this adds

A SystemInfo volume data source that projects the trust anchors of a named trust bundle to a PEM file:

volumes:
- name: trust
  systemInfo:
    dataSources:
    - trustBundle:
        name: egress-mitm.ate.dev
        path: egress-ca.pem

Inspired by the Kubernetes clusterTrustBundle projected volume source, but source-neutral: the template names a bundle; where it's fetched from is a deployment concern behind ateapi, not part of the API.

Design points

  • Allowlist in ateapi, not the CRD schema — today only egress-mitm.ate.dev (the egress gateway CA bundle, Add an SDS server that mints a short-lived leaf per SNI on demand at handshake time #823), mapped to the ClusterTrustBundle that atecontroller's EgressMITMTrustReconciler (Add EgressMITMTrustReconciler #946) derives from the egress-mitm-ca-pool Secret. The signer-linked object name stays a backend detail; the future backend registry widens the allowlist without an API change.
  • Resolution in ateapi at actor start, sanitized kubelet-style (internal/pemutil); the wire carries only {path, pem_bundle}; atelet writes bytes at a stable path with the temp+rename discipline from feat: add systemInfo volume source with actorIdentity data source #803 (find-paths safe) and never talks to any backend. Contents refresh on every Run/Restore.
  • Fail-closed, with startup that can tell its failure modes apart: unknown names, unusable bundles, and unavailable backends fail actor start naming the bundle. ateapi's boot probe is a one-shot authorized List — a genuinely unserved API degrades with a warning; stale RBAC fails startup naming the missing rule (rather than hanging cache sync); transient apiserver errors retry briefly then fail startup rather than silently degrading the replica.
  • Schema note: dataSources MaxItems tightened 32→8 while adding the trustBundle member. Vacuous in practice (the old schema couldn't admit more than one entry), but flagged since it's ratchet-shaped.

E2E — delivery and consumption

Delivery (identity suite, both sandbox classes): provisions the egress-mitm-ca-pool Secret and drives the real #946 reconciler (writing the bundle directly isn't possible — the reconciler reverts hand-edits), asserts the projected file byte-exact, then rotates the pool across a suspend/resume to prove refresh-on-restore. Since the probe fixture is shared and fail-closed, e2e.DeployProbe itself ensures the bundle exists for whatever suite deploys it.

Consumption (new egressmitm suite, both sandbox classes): deploys the sdsmint (MITM) egress gateway and proves an actor completes a TLS handshake with the gateway's per-SNI minted leaf using ONLY the projected anchors — plus a system-roots negative control that must fail. The pair is unambiguous in both directions: the positive can't pass under passthrough (the bundle holds no public CAs), and the negative can't fail under passthrough.

Not in this PR

@thompsonmax
Max Thompson (thompsonmax) force-pushed the ctb-source branch 2 times, most recently from 9cfd863 to eb71544 Compare August 20, 2026 20:00
@thompsonmax Max Thompson (thompsonmax) changed the title [stacked on #803] Add clusterTrustBundle as a SystemInfo volume data source Add clusterTrustBundle as a SystemInfo volume data source Aug 20, 2026
@thompsonmax Max Thompson (thompsonmax) changed the title Add clusterTrustBundle as a SystemInfo volume data source Add trustBundle as a SystemInfo volume data source Aug 20, 2026
@thompsonmax
Max Thompson (thompsonmax) marked this pull request as ready for review August 20, 2026 20:47
@thompsonmax
Add trustBundle as a SystemInfo volume data source
8f73149 
A trustBundle data source projects the trust anchors of a named trust
bundle to a PEM file in the volume — inspired by the Kubernetes
clusterTrustBundle projected volume source, but source-neutral: the
template names a bundle, and where it is fetched from is a deployment
concern behind ateapi, not part of the API (agent-substrate#932).

Supported names are allowlisted in ateapi rather than the CRD schema,
so the eventual configurable backend registry widens them without an
API change. Initially the only supported bundle is egress-mitm.ate.dev
(the egress gateway CA bundle, agent-substrate#823), resolved from the
ClusterTrustBundle that atecontroller's EgressMITMTrustReconciler
(agent-substrate#946) derives from the egress-mitm-ca-pool Secret — the signer-linked
object name (egress-mitm.ate.dev:mitm:primary-bundle) is a backend
detail the allowlist mapping keeps out of the template API. Resolution
goes through an informer-backed lister and sanitizes the way kubelet
sanitizes projections: CERTIFICATE blocks only, deduplicated, headers
stripped (internal/pemutil).

The wire spec carries only {path, pem_bundle} resolved bytes; atelet
writes them at a stable path with the per-file temp+rename discipline
(find-paths safe) and never talks to any bundle backend. Actor start
fails closed, naming the bundle, when the name is not allowlisted, the
backend is unavailable in this deployment (certificates.k8s.io/v1beta1
is feature-gated; ateapi probes at startup and degrades instead of
hanging), or the bundle is missing, empty, or free of certificates.

The identity e2e drives the real chain end to end: it provisions the
egress-mitm-ca-pool Secret, waits for the reconciler to publish the
derived bundle, asserts the projected file in both CI lanes, then
rotates the pool and asserts a resumed actor observes the new contents
at the same path — writing the bundle directly is not an option, since
the reconciler reverts or deletes hand-written contents. Bundle
contents refresh on every Run/Restore; live refresh for running actors
is agent-substrate#932 PR 2. ateapi gains RBAC to read
clustertrustbundles.certificates.k8s.io.
@thompsonmax
e2e: prove actor TLS through the MITM egress gateway with the project…
37df9b3 
…ed bundle

The identity suite verifies trust-anchor DELIVERY (pool -> reconciler ->
bundle -> resolution -> projection); this adds the CONSUMPTION half for
agent-substrate#871: an actor completes a TLS handshake with the sdsmint egress
gateway's per-SNI minted leaf using ONLY the anchors projected through
its trustBundle SystemInfo volume — on both sandbox classes, since
delivery differs per class (gVisor RO bind vs the micro-VM unified
virtio-fs share).

The probe gains /fetch?url=&roots=bundle|system, which GETs over the
actor's normal egress path with TLS roots from the projected bundle or
the image's system roots. The new egressmitm suite asserts the pair
that makes the result unambiguous: roots=bundle succeeds (and would
fail under a passthrough gateway, whose relayed public certificates the
bundle cannot validate — so a pass also certifies interception is on),
while roots=system fails certificate verification (the minted leaf
chains to no public CA; under passthrough it would succeed).

The sdsmint gateway variant replaces the passthrough gateway
cluster-wide, so CI deploys it as a separate step after both standard
lanes and runs only this suite against it — once per sandbox class,
gated by E2E_EGRESS_MITM. The suite ensures (never replaces) the CA
pool: sdsmintd signs with the pool mounted into the gateway pod, and
replacing it would race kubelet's Secret propagation into that mount.
@thompsonmax Max Thompson (thompsonmax) mentioned this pull request Aug 20, 2026
Draft
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@bowei Bowei Du (bowei) Awaiting requested review from bowei
@ahmedtd Taahir Ahmed (ahmedtd) Awaiting requested review from ahmedtd
@haiyanmeng haiyanmeng Awaiting requested review from haiyanmeng

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@thompsonmax