Skip to content

feat: add structured recall evaluation and execution traces - #5

Merged
itxaiohanglover merged 2 commits into
open-agent-power:mainfrom
FloretKu:agent/structured-recall
Jul 30, 2026
Merged

feat: add structured recall evaluation and execution traces#5
itxaiohanglover merged 2 commits into
open-agent-power:mainfrom
FloretKu:agent/structured-recall

Conversation

@FloretKu

@FloretKu FloretKu commented Jul 26, 2026

Copy link
Copy Markdown
Contributor

Summary

  • add versioned, goal-aware JSON recall output with score explanations and correct pre-limit type filtering
  • add deterministic offline recall evaluation with Recall@1/3/5, MRR, nDCG@5, leakage/no-result rates, latency, run manifests, snapshots, and run comparison/Goal lift
  • add append-only execution traces, judge/human feedback, blockers, validated result states, and review-only Wiki/Skill proposals
  • replace the stale frontmatter contract, add formal dataset/trace/manifest schemas, remove model credentials from the core default config, and strengthen health checks

Safety boundaries

  • recall evaluation is read-only and fails if persistent KB state changes
  • trace payloads reject credential-like fields
  • proposals only write to drafts/proposals/; they never modify formal Wiki/Skill content
  • the example dataset is format-only and is not presented as human-labeled evidence

Verification

  • 43 passed
  • Python compileall passed
  • git diff --check passed
  • wheel build passed; wheel contents include evaluation/trace/proposal modules and all three new schemas

Still required for experiment claims

  • 30-50 human-reviewed recall cases
  • external Judge/model decision
  • fixed GitHub task set, sandbox repositories, execution model, and token budget for the four-way ablation

@FloretKu FloretKu changed the title feat: add structured goal-aware recall output feat: add structured recall evaluation and execution traces Jul 26, 2026
@itxaiohanglover
itxaiohanglover marked this pull request as ready for review July 29, 2026 13:26
@itxaiohanglover
itxaiohanglover requested a review from MargoGuo July 29, 2026 13:26
@itxaiohanglover
itxaiohanglover merged commit 15e57ca into open-agent-power:main Jul 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants