Skip to content

feat(curator): add governed maintenance and accuracy evaluation - #92

Closed
humandebri wants to merge 2 commits into
mainfrom
feat/curator-v1
Closed

feat(curator): add governed maintenance and accuracy evaluation#92
humandebri wants to merge 2 commits into
mainfrom
feat/curator-v1

Conversation

@humandebri

@humandebri humandebri commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator

Purpose

  • Add a governed Curator workflow for maintaining the Memory, Knowledge, Skills, and Sessions stores without allowing an agent to mutate wiki content directly.
  • Provide a reproducible, AI-only staging accuracy and safety evaluation workflow while keeping the existing staging database read-only.

Changes

  • Add curator scan, strict kinic.curator.plan.v1 validation, dry-run-first proposal selection, etag revalidation, and one atomic batch only when --confirm is supplied.
  • Keep /Sources read-only and reject move, delete, retry, overwrite, cross-store mixing, low-confidence proposals, duplicate targets, and batches over 100 operations.
  • Add curator_accuracy_eval for answer-masked annotation input, two-run annotation comparison, explicit third-run adjudication, deterministic and semantic scoring, and proposal recall enforcement.
  • Add a reusable 128-node controlled staging seed across all four mutable stores and /Sources, with strict manifest validation and exact batch validation immediately before every write.
  • Split accuracy report v2 into controlled_seed and staging_existing cohorts so synthetic fixtures cannot satisfy the real-data sample threshold.
  • Add the kinic-wiki-curator skill and update CLI, staging, and accuracy documentation. Curator artifacts remain private and ignored by Git.

Verification

  • ./.local/check.sh
  • cargo test -p kinic-vfs-cli curator_accuracy --locked (22 accuracy and seed tests passed)
  • cargo clippy -p kinic-vfs-cli --all-targets --all-features -- -D warnings
  • cargo fmt --all -- --check
  • node scripts/check-docs-links.mjs (24 files passed)
  • bash -n scripts/staging/seed_curator_accuracy.sh
  • git diff --check
  • CLI help contracts for kinic-vfs-cli curator and curator_accuracy_eval
  • Retained seed definition validation passed for 197 total nodes, including 128 non-folder cases.

Staging Evaluation

  • No canister deployment or upgrade was performed because this change adds no Wasm, Candid, schema, or migration changes.
  • No Curator proposal was confirmed or applied to the retained staging database.
  • The controlled deterministic cohort contains 128 nodes and passes its regression thresholds.
  • The pre-existing staging cohort contains only 7 nodes and is correctly reported as insufficient_sample; the result is not production accuracy evidence.
  • The retained semantic labels require proposals for 12 findings, while the old plan contains proposals for only the 6 duplicate findings. Report v2 therefore rejects that plan with proposal recall 0.5; a fresh same-snapshot semantic annotation and plan are required before claiming a semantic pass.

Impact

  • Introduces CLI and skill surfaces without changing existing store, search, ranking, publication, or query_context behavior.
  • Remote writes remain explicitly approved, etag-guarded, limited to seed-owned or selected mutable-store paths, and atomic.
  • Accuracy artifacts contain private wiki content and must remain mode 0600 under the ignored .benchmarks/ tree.

Add strict AI annotation, proposal recall, cohort scoring, and reusable controlled seed tooling. Validate retained seed manifests and exact write batches before any staging mutation.
@humandebri humandebri changed the title feat(curator): add governed wiki maintenance workflow feat(curator): add governed maintenance and accuracy evaluation Aug 14, 2026
@humandebri humandebri closed this Aug 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant