feat(cloud): publish archives with explicit admission evidence

[vincentkoc]

Additional instructions

This is a same-repository branch; maintainers retain normal push access.
The fork-only "Allow edits from maintainers" checkbox does not apply.

What Problem This Solves

Operators need to distinguish an intact archive with incomplete or unknown
source/enrichment evidence from a fully enriched archive before publication.
The existing strict default remains unchanged.

Why This Change Was Made

Add explicit cloud publish --admission-policy=archive-v1 --observation-order.
The frozen SQLite snapshot records deterministic source, integrity, and
enrichment evidence. Its canonical warnings travel with the snapshot and must
match during publication, reader verification, and replay.

Archive admission retains schema, required-data, referential-closure, full-body,
privacy, and size checks. Lossy portable exports are rejected before upload.
The remote must explicitly advertise both archive-admission and
observation-order support; this PR does not deploy or activate remote support.

Use the normally released Crawlkit v0.15.1 optional warning fields, without
local replacements or schema/wire-version changes.

User Impact

• Opted-in publications retain truthful incomplete or unknown evidence instead
  of silently presenting it as complete.
• Default strict publication, canonical contents, privacy, and existing limits
  are preserved.
• Existing stored provider IDs remain unknown where applicable and are not
  rewritten or repaired.

Evidence

• Published Crawlkit v0.15.1 source and Go checksums verified.
• Focused CLI proof compiled against that release. The initial run exposed
  three fixture defects; its failure is retained.
• Reviewed fixture corrections passed all three affected test families and
  13 subtests, with no skips. Tests used isolated local caches, readonly modules,
  denied external networking, and loopback fixtures.
• Other unchanged focused admission, identity, privacy, revision-coverage,
  store-observation, and portable-export checks retain their prior passing
  evidence.
• Formatting and diff checks pass. Full Linux/macOS tests, coverage, build,
  smoke, and snapshot checks remain gated on this PR's native CI.
• Independent full-source review passed. Local autoreview stopped before
  invoking a reviewer because its secret scanner misclassifies the unchanged
  Go expression "mutation_token": mutationToken in diff context. It produced
  no native review verdict. Publication has an explicit maintainer exception
  for that false positive; exact-head hosted ClawSweeper, CI, and code-owner
  clearance remain required before merge. The scanner was not disabled or
  modified, and the review input was not filtered.
• Production delta: +677/-37, implementing the approved source-observation and
  immutable archive-admission contract. Tests: +757/-9. The final dependency
  and fixture continuation adds no production code.

No production export, live archive publication, binary installation, or
historical-data repair was performed.

Named follow-up: teach the canonical openclaw/agent-skills autoreview secret
scanner to recognize ordinary Go variable references without weakening
credential detection. That shared-tool repair is outside this PR.

[clawsweeper]

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

[clawsweeper]

Codex review: needs maintainer review before merge. Reviewed September 9, 2026, 1:40 PM ET / 17:40 UTC (Revision 2).

ClawSweeper review

What this changes

Adds opt-in cloud archive publication with frozen integrity, source-observation, and enrichment evidence, carrying immutable warnings through staging, replay, and reader verification.

Merge readiness

✅ Ready for maintainer review

This remains a useful, distinct contribution absent from current main and v0.9.5. No blocking introduced defect was found; the earlier review’s empty findings remain appropriate.

Priority: P2
Reviewed head: e2d1dd402c2740e162d5dcc0997886b77b9f512b

Review scores

Measure	Result	What it means
Overall readiness	🐚 platinum hermit (4/6)	A coherent opt-in implementation with substantial focused coverage and no identified blocking defect.
Proof confidence	🌊 off-meta tidepool	Not applicable: The MEMBER-authored PR is exempt from ordinary contributor proof; inspected fixtures exercise snapshot preparation, publication, replay, and warning verification, but do not establish live Worker publication.
Patch quality	🐚 platinum hermit (4/6)	No actionable review findings were identified.

Verification

Check	Result	Evidence
Real behavior	Not applicable	Not applicable: The MEMBER-authored PR is exempt from ordinary contributor proof; inspected fixtures exercise snapshot preparation, publication, replay, and warning verification, but do not establish live Worker publication.
Evidence reviewed	8 items	Verified scope and continuity: The supplied introduction evidence pins the 14-file delta from dab4d84 to e2d1dd4. The previous completed review examined this same head and recorded no findings. Local comparison confirmed the admission files remain unchanged. Still necessary on main and release: Current main and the inspected v0.9.5 snapshot builder expose strict publication and allow-incomplete, without the new admission assessment or immutable warnings. The open-PR listing returned only this PR. Artifact compatibility coverage: TestArchiveAdmissionFreezesDeterministicEvidence checks unchanged source bytes, deterministic digests, retained canonical content, removal of stale markers, rejection of mutated assessments, unchanged strict rejection, and preserved allow-incomplete behavior. The added metadata is written only into the sanitized copy.
Findings	None	None.
Security	None	None.

How this fits together

Gitcrawl’s cloud publisher turns a local SQLite archive into a sanitized snapshot and sends it to a separately deployed Worker service. The new admission mode records archive limitations before publication while preserving existing authentication and strict defaults.

flowchart TD
  A[Local SQLite archive] --> B[Freeze archive]
  B --> C[Assess source and enrichment]
  C --> D[Sanitize and validate integrity]
  D --> E[Store evidence and hash snapshot]
  E --> F[Check remote capabilities and roles]
  F --> G[Upload and stage]
  G --> H[Optional cutover and reader verification]

Loading

Before merge

None.

Agent review details

Security

None.

Review metrics

Metric	Value	Why it matters
Production and test delta	Production +677/-37; tests +757/-9	Production growth implements the stated source-assessment and immutable-admission contract, with focused compatibility and integrity coverage.

Technical review

Best possible solution:

Retain explicit admission opt-in, truthful frozen evidence, and capability-gated publication while preserving the existing strict path.

Do we have a high-confidence way to reproduce the issue?

Not applicable: this adds an explicit publication mode rather than repairing a reported failure; its behavior is covered by inspected source and focused test cases.

Is this the best way to solve the issue?

Yes: extending the existing snapshot pipeline avoids a parallel publisher and binds evidence to sanitized bytes while retaining strict defaults.

AGENTS.md: not found in the target repository.

Codex review notes: model internal, reasoning medium; reviewed against dab4d849f956.

Labels

Label justifications:

• P2: This is a bounded, opt-in archive publication improvement with preserved defaults.
• rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🌊 off-meta tidepool and patch quality is 🐚 platinum hermit.
• status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Not applicable: The MEMBER-authored PR is exempt from ordinary contributor proof; inspected fixtures exercise snapshot preparation, publication, replay, and warning verification, but do not establish live Worker publication.

Evidence

What I checked:

• Verified scope and continuity: The supplied introduction evidence pins the 14-file delta from dab4d84 to e2d1dd4. The previous completed review examined this same head and recorded no findings. Local comparison confirmed the admission files remain unchanged. (e2d1dd402c27)
• Still necessary on main and release: Current main and the inspected v0.9.5 snapshot builder expose strict publication and allow-incomplete, without the new admission assessment or immutable warnings. The open-PR listing returned only this PR. (internal/cli/cloud_snapshot.go:40, dab4d849f956)
• Artifact compatibility coverage: TestArchiveAdmissionFreezesDeterministicEvidence checks unchanged source bytes, deterministic digests, retained canonical content, removal of stale markers, rejection of mutated assessments, unchanged strict rejection, and preserved allow-incomplete behavior. The added metadata is written only into the sanitized copy. (internal/cli/cloud_admission_test.go:117, e2d1dd402c27)
• Publication and authorization boundary: The publisher checks requested remote capabilities and existing publisher/reader roles before upload. Admission changes archive eligibility, not credentials, principal selection, or authorization lifetime. Publisher and reader verification compare immutable snapshot warnings. (internal/cli/cloud_commands.go:161, e2d1dd402c27)
• Affirmative Crawlkit dependency: go.mod upgrades Crawlkit to v0.15.1, and cloud_snapshot.go directly populates its IngestManifest.Warnings field; cloud_commands.go reads ArchiveSnapshot.Warnings. This establishes the relevant dependency contract. (go.mod:12, e2d1dd402c27)
• Released optional warning fields: GitHub resolves v0.15.1 to d4efff69f0aaca172abcc3505a492024f30a9a7f. The runtime delta adds optional warnings fields to archive snapshots and ingest manifests; inspected compatibility tests preserve legacy JSON when warnings are absent and exercise warning transport. (remote/warnings_test.go:13, d4efff69f0aa)

Likely related people:

• unknown: The claimed source-line change could not be verified from bounded local history. (role: source history unknown; confidence: low)
• unknown: The claimed source-line change could not be verified from bounded local history. (role: source history unknown; confidence: low)

Rating scale

Score	Internal tier	Crab rank	Meaning
6/6	S	🦀 challenger crab	Exceptional readiness
5/6	A	🦞 diamond lobster	Very strong readiness
4/6	B	🐚 platinum hermit	Good normal PR; ordinary maintainer review
3/6	C	🦐 gold shrimp	Useful, but confidence is limited
2/6	D	🦪 silver shellfish	Proof or implementation needs work
1/6	F	🧂 unranked krab	Not merge-ready
N/A	NA	🌊 off-meta tidepool	Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (1 earlier review cycle)

• reviewed 2026-09-09T17:28:32.987Z sha e2d1dd4 :: needs maintainer review before merge. :: none