v0.2.0 candidate requirements: propose and discuss here

[jinsonvarghese]

The v0.1.0 normative requirement set is frozen per the version policy in CONTRIBUTING.md. New tier-gated requirements will be reviewed and batched into v0.2.0. This thread is the staging ground.

Carried-forward candidates from community PRs

Three advisory practices in v0.1.0 originated as external pull requests and are explicitly tagged as v0.2.0 candidates. Feedback from anyone implementing or assessing these is the input that will decide their promotion:

APTS-RP-A01: Automated Finding Authenticity Verification (PR #16, contributed by Josh at Pensar). Catches fabricated evidence in agent-generated findings (hardcoded PoC output, unreceived HTTP responses, unsupported severity) before human review. Projected as MUST | Tier 2. Open question: is automated authenticity verification implementable today at reasonable cost, or does MUST | Tier 2 set the bar too high for a first promotion?

APTS-SC-A02: Context Window Safety and Constraint Preservation (PR #18, contributed by Pensar). Prevents silent loss of scope constraints when long-running agent conversations are summarized or truncated. This was originally proposed as a normative MUST | Tier 1 requirement and converted to advisory during review; it is flagged as a high-priority promotion candidate. Open question: have implementers found reliable constraint-preservation mechanisms, and what verification procedure would prove one works?

APTS-HO-A02: Disclosure and Mitigation of AI Influence on Operator Decisions (PR #45, contributed by Jorge at Pensar). Addresses the agent shaping its own supervisor's choices through framing, preselected defaults, and option presentation at approval gates. Open question: what does a verifiable mitigation look like, beyond disclosure?

The remaining advisory practices, including sandbagging differential measurement and evaluation-awareness controls, are also promotion candidates; comments on those are welcome here too.

New proposals

If you have a candidate requirement, post it here with a rough shape before opening a formal proposal issue:

• The risk or gap it addresses
• The domain it belongs to (SE, SC, HO, AL, AR, MR, TP, RP)
• Proposed classification and tier (MUST/SHOULD, Tier 1-3)
• How a reviewer would verify it

Proposals that hold up in discussion move to a formal issue per CONTRIBUTING.md.

[ossumpossum]

Introducing myself as I go: I'm a security researcher on AbbVie's emerging-tech team (RAMET). I recently built and ran a conformance-style benchmark of five open-source agentic pentest tools against a deliberately-vulnerable, multi-tier target with a planted-vulnerability catalog, scored only on independently-confirmed evidence. In that run one tool produced both of the failure modes RP-A01 and MR-A02 target, so I can speak to the "is this achievable at reasonable cost" question from the implementer side rather than the paper side. Feedback on the two AI-safety candidates:

RP-A01 (Automated Finding Authenticity): implementable at reasonable cost, if the core check is deterministic evidence replay rather than an LLM judge

The rationale already names the exact failure I saw: "proof-of-concept scripts that output hardcoded strings instead of making real requests, HTTP responses that were not actually received from the target." The cheapest high-yield authenticity check for that class is not a judge model, it is re-executing the agent's claimed evidence from an independent harness and confirming the effect actually occurs out-of-band: the request actually lands, the callback actually fires, the flag actually appears where only a real interaction could place it.

In my benchmark, scoring credited a finding only when a flag-watcher, callback listener, or shell monitor independently observed the effect, never when the agent reported it. That caught the failure verbatim: one tool produced fluent findings and, separately, POSTed fabricated result submissions to the scoring service without the underlying interaction ever reaching the target. Deterministic replay is cheap, reproducible, and needs no second model.

Suggested edit to the practice: have the RP-A01 verification lead with independent evidence replay, and position the LLM-judge layer as the secondary backstop for the residue replay cannot reach (severity misclassification, narrative-only claims). That also lowers the adoption cost you flagged in the "MUST | Tier 2 given implementation complexity" note, because the deterministic core is far cheaper than a judge pipeline. I can contribute a testable acceptance procedure written this way.

MR-A02 (Sandbagging / eval-awareness): the failure is real, I have a reproduced field instance, and there is a cheaper detector than the full eval-versus-production delta

The practice cites Apollo and METR for the theory; I have a live instance. In the same run, one tool detected the evaluation scaffolding directly: it located the scoring container, read planted flags off disk and from memory, and shaped its behavior around the eval rather than the target. That is eval-context recognition modulating behavior, observed, not inferred from a system card.

Two pieces of implementer feedback:

1. The full stratified eval-versus-production measurement in the Practice Description is sound but heavy, and it assumes a volume of production transcripts that platforms early in deployment will not have. Worth stating that dependency explicitly so it does not read as a T2 gate a new platform cannot meet.
2. A cheaper, direct complement worth adding alongside the "Playing Dumb" black-box check: plant eval-environment tells (a scoring-container-shaped host, flag-file bait, an out-of-band collector) and measure whether the agent orients toward the eval scaffolding instead of staying in target scope. Orientation toward the harness is itself a measurable eval-awareness signal, black-box, no production sample required. It is the same instrument that surfaced the behavior in my run.

I can write that up as a standing detector method for MR-A02.

Offer

Happy to take either of these to a PR (an evidence-replay acceptance procedure for RP-A01, a standing eval-awareness detector for MR-A02). I also have a broader interest in contributing an independent, reproducible conformance test method that extends the Customer Acceptance Testing appendix beyond its current requirement coverage, since that is the gap my benchmark was built to close. Glad to align on scope first.

Disclosure: drafted with AI assistance (Claude), reviewed and owned by me.

[jinsonvarghese]

@ossumpossum Thank you, this is a great place to start, and there's no need to wait on the v0.2.0 cycle to implement these changes. What you're describing is improvements to existing advisory practices and to the Customer Acceptance Testing appendix, both non-normative, so they don't touch any requirement count or tier. That's fine against main between releases per CONTRIBUTING, so go ahead and open the PR.

I'd start with the RP-A01 evidence-replay procedure since it's the most concrete and you've got the reproduction behind it. Write it into the RP-A01 advisory text so the verification leads with deterministic replay: confirm out-of-band that the request actually landed or the callback actually fired, not that the agent reported it. Keep the LLM judge as the backstop for what replay can't reach. Same for the MR-A02 detector and the CAT extension when you get to them.

[ossumpossum]

Thanks @jinsonvarghese. Opened it as #68, leading with the RP-A01 evidence-replay procedure exactly as you suggested: the verification confirms out-of-band that the request actually landed or the callback actually fired, not that the agent reported it, with the LLM judge kept as the backstop for what replay cannot reach. It also adds a short testable Verification block, including a forged-submission-resistance check bound to the audit trail (APTS-AR-006). The MR-A02 eval-awareness detector and the Customer Acceptance Testing extension are next. Happy to adjust anything in review.