Skip to content

report.md

Latest commit

 

History

History
17 lines (13 loc) · 850 Bytes

report.md

File metadata and controls

17 lines (13 loc) · 850 Bytes
Raw

FlakeWarden evaluation report

  • Corpus size: 150 labeled failures
  • Overall accuracy: 90.7%
  • Safety false-positive rate (real defect hidden as flaky/environment): 0.0% (0 cases) — target ~0%
  • Noise false-alarm rate (flaky/environment re-escalated as defect): 12.0% (12 cases)
  • Routing: 52 resolved by the deterministic scorer, 98 escalated to the grounded classifier, 52 routed to the human-review gate

Confusion matrix

actual \ predicted real_defect flaky environment
real_defect 50 0 0
flaky 12 43 0
environment 0 2 43

Generated by python eval/harness.py --report eval/report.md. Backend: offline rule-based classifier (deterministic, no API key). With ANTHROPIC_API_KEY set, the grounded Claude classifier handles the ambiguous band.