Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
39 commits
Select commit Hold shift + click to select a range
03a96de
feat(plugin): 1/6 — marketplace + coder-eval plugin skeleton
uipreliga Aug 4, 2026
aa14c66
feat(plugin): 2/6 — generate the bundled criteria reference, guard it…
uipreliga Aug 4, 2026
0882960
feat(plugin): 3/6 — skill-check skill and the canonical activation suite
uipreliga Aug 4, 2026
8d06e3d
feat(plugin): 4/6 — init and task skills
uipreliga Aug 4, 2026
828551d
feat(plugin): 5/6 — analyze and ci skills
uipreliga Aug 4, 2026
d33e259
feat(plugin): 6/6 — validate the plugin in CI, extend CE026, document it
uipreliga Aug 4, 2026
80ff825
fix: code review fixes for the Claude Code plugin marketplace
uipreliga Aug 5, 2026
bcc2d87
test: defer the plugin-skill repo-file containment guard
uipreliga Aug 5, 2026
004eae6
fix(plugin): name the tautological-criterion trap in init and task
uipreliga Aug 5, 2026
e6468f3
fix(plugin): make analyze compute its numbers, and weight smoke crite…
uipreliga Aug 5, 2026
41137c2
feat(lint): CE026 clause 4 — snippet `with:` keys must be real action…
uipreliga Aug 5, 2026
00ab354
feat(plugin): 1/5 — the bundled criteria reference explains optional …
uipreliga Aug 5, 2026
6983147
feat(plugin): 2/5 — shared adversarial task rubric, applied by `task`
uipreliga Aug 5, 2026
8162f69
feat(plugin): 3/5 — a real run becomes part of done in `task`
uipreliga Aug 5, 2026
9543df4
feat(plugin): 4/5 — `/coder-eval:lint-tasks`, a read-only reviewer of…
uipreliga Aug 5, 2026
3c1dac5
feat(plugin): 5/5 — activation budgets in `skill-check`, layer routin…
uipreliga Aug 5, 2026
27198b0
fix: make lint-tasks' read-only rule outlive the frontmatter deny
uipreliga Aug 5, 2026
68e87a1
docs: defer one harness candidate from the plugin-audit run
uipreliga Aug 5, 2026
8c8811e
fix: code review fixes for the plugin-audit P0/P1 plan
uipreliga Aug 5, 2026
9c3c767
fix(tasks): armed positives must require success, guarded by CE034
uipreliga Aug 5, 2026
3b7bfbe
fix(plugin): address PR #82 review — reachable activation suites, lea…
uipreliga Aug 5, 2026
db5f53a
docs: add Tutorial 07 for the plugin, and fix two gaps in PLUGIN.md
uipreliga Aug 5, 2026
32f8586
docs: rework Tutorial 07 after review — accuracy fixes and far less n…
uipreliga Aug 5, 2026
9d3f0b8
feat(plugin): CLI-driving skills offer to install coder-eval, asking …
uipreliga Aug 5, 2026
0ac1f3a
fix: reconcile the plugin branch with main after rebase
uipreliga Aug 10, 2026
87f1cb7
fix(plugin): address the three PR #82 findings left open
uipreliga Aug 10, 2026
a0b2ff5
refactor: retire the repo-local twins of the plugin's authoring skills
uipreliga Aug 11, 2026
019738b
feat(plugin): 1/6 — discover the eval tree instead of assuming tasks/…
uipreliga Aug 11, 2026
89dc8e0
feat(plugin): 2/6 — analyze reads the run's actual schema, not one ge…
uipreliga Aug 11, 2026
d5d824b
feat(plugin): 3/6 — resolve the version a project pins before validat…
uipreliga Aug 11, 2026
a03b144
feat(plugin): 4/6 — look before you write: skill-check, init, lint-tasks
uipreliga Aug 11, 2026
9950a28
feat(plugin): 5/6 — repo convention wins, and the CI gate stops measu…
uipreliga Aug 11, 2026
bc14b3a
feat(plugin): 6/6 — document the criterion aliases the loader accepts…
uipreliga Aug 12, 2026
d4ef5cd
fix: code review fixes for the plugin generic-adopter plan
uipreliga Aug 12, 2026
cccb590
test(plugin): make a new skill declare whether it needs eval-root dis…
uipreliga Aug 12, 2026
17162c4
refactor(plugin)!: rename skill-check to check-skill, and write the n…
uipreliga Aug 12, 2026
7a84436
docs: stop teaching the recursive task glob the ci skill forbids
uipreliga Aug 12, 2026
6c16377
fix(plugin): wire the skill source into the ci skill's scheduled drif…
uipreliga Aug 12, 2026
17491f0
chore(plugin): pin plugin.json to 0.9.6 after rebasing onto the release
uipreliga Aug 12, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions .claude-plugin/marketplace.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
{
"name": "coder-eval",
"owner": {
"name": "UiPath",
"url": "https://github.com/UiPath/coder_eval"
},
"description": "Evaluate and benchmark AI coding agents and Claude Code skills.",
"plugins": [
{
"name": "coder-eval",
"source": "./plugins/coder-eval",
"description": "Author, run, and analyze coder-eval suites — including whether your Claude Code skills actually trigger.",
"category": "testing",
"keywords": ["evaluation", "testing", "skills", "benchmark", "ci"]
}
]
}
2 changes: 1 addition & 1 deletion .claude/commands/coder-eval-review.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ description: Generate per-task review.json (summary + tags) for a completed run

## Context

If `$ARGUMENTS` is empty or blank, default to `runs/latest` and inform the user: "No path provided — reviewing the latest run at `runs/latest`."
If `$ARGUMENTS` is empty or blank, resolve the run to review per the "No target given" rule in `.claude/shared/run-layout.md` — discover the run root, then take `latest` if that symlink resolves and the newest run directory by name if it does not. Tell the user which run you picked and how you got there before reading anything.

You are producing post-run reviews for a coder_eval evaluation run. The target path is: `$ARGUMENTS`

Expand Down
178 changes: 0 additions & 178 deletions .claude/commands/coder-eval-run-analysis.md

This file was deleted.

Loading
Loading