G1: fence the ingest→ACL chain at the pipeline seam, mark G1/G2/G9 shipped

[brentrager]

The finding

I was sent to build G1 — the Connector trait, a MockConnector, and an ingest→chunk→embed→store contract test. All of it already exists on main and has since 315841c7: the seam in rust/ingestion/src/connector.rs, the mock, the file / web / github connectors, the chunker (G2), and incremental indexing.

docs/Planning/Feature Gaps.md was never updated to say so. §G1, §G2 and §G9 still read as open gaps while §G3 and §G8 carry ✅ markers — which is how an agent gets dispatched to rebuild shipped code. Most of this PR is correcting that record.

The one real hole underneath the stale doc

The guarantee that a connector's ACL survives ingestion — RawDocument::acl → chunk → structured DocAcl → AclKnowledgeStore side table → AclReader — was asserted end to end in exactly one place: github_connector.rs::custom_group_gates_retrieval_end_to_end.

That is a connector test. Rewrite or delete the GitHub connector and the ingest half of G3 loses its only fence, silently, with ingestion_contract.rs still green.

ingested_acls_gate_retrieval_for_every_connector asserts the same chain at the pipeline seam over a MockConnector, so it binds every connector present and future:

• a doc ingested for group-eng is readable by a principal carrying that group, and returns nothing for group-fin or for anonymous;
• symmetrically for a group-fin doc;
• a doc ingested with no ACL stays org-public — so the gate is an opt-in restriction, not a blanket denial that would make the leak assertions pass for the wrong reason.

Every negative assertion is paired with the entitled-principal positive control on the same query, plus a run-level documents_pulled == 3 / chunks_stored >= 3. "Nothing leaked" must not be satisfiable by a pipeline that stored nothing.

Red before green

Reverted DocAcl::for_groups(acl.clone()).attach_to(document) in pipeline.rs, in place, and re-ran:

running 2 tests
test ingest_chunks_embeds_stores_and_retrieves_then_is_idempotent ... ok
test ingested_acls_gate_retrieval_for_every_connector ... FAILED

thread 'ingested_acls_gate_retrieval_for_every_connector' panicked at
ingestion/tests/ingestion_contract.rs:229:5:
group-fin must not read the group-eng doc, got 1 hits: ["The snarflex incident
postmortem is engineering-only. Snarflex retries were exhausted before the pager
fired."]

test result: FAILED. 1 passed; 1 failed

That is the exact G3 cross-user leak. Note the pre-existing contract test stayed green through the revert — which is precisely what kept this gap invisible.

Restored; full suite green (53 tests: 41 + 6 + 3 + 2 + 1).

What the ✅ markers deliberately do not cover

Recorded in the doc rather than glossed:

• pull returns Vec<RawDocument>, not the planned Stream<Document>. It materializes a whole source per pull. Fine for file/web/github, wrong for a large Confluence space or Jira project — which is exactly what comes next. Re-shaping it is a breaking trait change, so it belongs before the SaaS connectors, not after.
• No format extraction (PDF/DOCX/rich HTML → text) ahead of the chunker. The GitHub connector skips binary extensions rather than reading them.
• .github/workflows/rust.yml has no schedule:, so G9's gated external tier (#[ignore] + SMOOTH_AGENT_E2E) is a convention nothing ever executes. A live-API break in the web or GitHub connector is invisible until a user hits it.
• The connector long tail — confluence, jira, notion, slack, zendesk, drive, salesforce, sharepoint. Their per-document permissions must map onto RawDocument::acl the way GithubConnectorConfig::acl_groups already does, or ingesting them reopens G3.

Verification

• cargo test -p smooai-smooth-operator-ingestion — 53 passed, 0 failed, 1 ignored (the network-gated web fetch)
• cargo clippy -p smooai-smooth-operator-ingestion --all-targets — clean
• cargo fmt --all -- --check — clean

No production behavior changes: one test, one changeset, one doc.

[changeset-bot]

🦋 Changeset detected

Latest commit: f065025

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 2 packages

Name	Type
@smooai/smooth-operator	Patch
@smooai/smooth-operator-web-chat-example	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR