Skip to content

Add unquantified check for vague answers to counting prompts - #42

Merged
royalpinto007 merged 4 commits into
AgentPostmortem:mainfrom
HarshRajSinghania:feat/unquantified-check
Sep 11, 2026
Merged

Add unquantified check for vague answers to counting prompts#42
royalpinto007 merged 4 commits into
AgentPostmortem:mainfrom
HarshRajSinghania:feat/unquantified-check

Conversation

@HarshRajSinghania

Copy link
Copy Markdown
Contributor

Summary

Adds a new unquantified check that flags results which answer a counting/listing prompt with a vague quantifier and no countable output.

Motivation

Fixes #41. A prompt such as "report every failing test as node ids" answered with "several hot paths worth investigating" currently produces no finding.

Implementation

check_unquantified in agentrace/checks.py:

  • Fires only when the prompt looks like a count/list/enumerate request (report every, how many, enumerate, etc.).
  • Skips if the result already has digits, markdown bullets, or table rows.
  • Flags low severity when the result still uses a vague quantifier (several, some, a few, many, ...).

Registered next to the other checks in CHECKS. README table updated.

Testing

Ran locally:

python3 -m pytest tests/ -q

Result: 45 passed.

New tests cover the issue's two acceptance cases:

  • counting prompt + "there are several issues" → flagged
  • same prompt + "3 failing: a, b, c" → clean

@github-actions

Copy link
Copy Markdown

Welcome, @HarshRajSinghania, and thanks for your first pull request to Agentrace.

A quick look at CONTRIBUTING.md covers how things are set up and run here.

I'll review this shortly. Ask anything in the thread if something is unclear.

@royalpinto007
royalpinto007 merged commit decd861 into AgentPostmortem:main Sep 11, 2026
8 checks passed
@github-actions

Copy link
Copy Markdown

Nice work @HarshRajSinghania, this is merged.

Good, focused change. Exactly the kind of thing Agentrace needs.

If Agentrace is useful to you, a star goes a long way.

Happy to see more from you whenever you have the time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

New check unquantified: flag vague answers to counting prompts

2 participants