RFC: Agent Outcome Observability and Assurance Metrics

[dinpd]

Publication status

Published on main in PR #59: Agent Outcome Observability and Assurance Metrics, community draft 0.1.

The publication acceptance checks are complete. This issue remains open as the community feedback and revision thread for the experimental draft.

Opportunity

Publish a companion RFC to the Agent Action Boundary Evidence proposal that defines how action evidence may be assessed and aggregated into meaningful outcome, constraint, control-effectiveness, execution-discipline, and evidence-quality measurements.

The boundary-evidence RFC answers what claims and identifiers exist. This RFC should answer what counts as a good, bad, prevented, missed, or indeterminate result, and how compatible results may be aggregated.

User and community impact

Agent observability systems often collapse execution success, policy decisions, outcome quality, and safety into one score. That can count a prevented harmful action as a failure, treat a successful but noncompliant execution as success, hide missing evidence, or compare unlike jobs. A public measurement contract gives operators, evaluators, and observability vendors a precise vocabulary for exchanging and reviewing these results.

Scope

• Relationship to the Agent Action Boundary Evidence RFC and OpenTelemetry causal telemetry.
• Separate measurement dimensions for intent fulfillment, constraint compliance, control effectiveness, control friction, execution discipline, and evidence quality.
• Outcome and adjudication states, including explicit indeterminate and excluded states.
• Versioned measurement profiles, eligibility rules, denominators, exclusions, and comparability boundaries.
• Derived metric definitions and anti-gaming requirements.
• Evidence confidence, provenance, freshness, and completeness.
• Privacy, cardinality, retention, and tenant-isolation guidance.
• Representative conformance scenarios and open questions for community feedback.

Non-goals

• A universal agent score or cross-profile leaderboard.
• Treating trace data, gate decisions, or model self-reports as ground truth.
• Standardizing a specific telemetry backend, database, evaluator, or UI.
• Capturing chain-of-thought or unrestricted raw prompts/tool payloads.
• Redefining the canonical action evidence envelope from the boundary-evidence RFC.

Acceptance criteria

• A new proposal document is published under docs/proposals/ and linked from docs/proposals/README.md.
• The RFC explicitly depends on, but does not redefine, the Agent Action Boundary Evidence proposal.
• It distinguishes execution outcome, intent fulfillment, constraint compliance, control effectiveness, control friction, and evidence quality.
• It defines safe/unsafe × allowed/intervened measurement semantics without assuming the gate decision is ground truth.
• It defines eligible populations, denominators, exclusions, indeterminate results, profile/version comparability, and derived metrics.
• It prohibits silent exclusion of missing or failed work and a universal cross-profile agent score.
• It includes privacy/cardinality guidance and an OpenTelemetry mapping that keeps durable assessment evidence independent of trace sampling.
• It includes representative conformance scenarios and focused questions for external reviewers.

Acceptance-test plan

Automated checks:

• Verify the RFC file and proposal-index link exist.
• Verify required normative sections and metric terms are present.
• Check Markdown links and repository formatting conventions.
• Run the repository's existing documentation-adjacent and test checks in proportion to this docs-only change.
• Review the final diff for secrets, personal data, unsafe telemetry examples, and accidental scope.

No human-only validation is required for publication; technical and community feedback will occur on this issue and the pull request.

Risks and security considerations

• Metrics can be gamed when eligibility or exclusions are underspecified.
• Gate decisions must not be treated as independent ground truth.
• Raw prompts, tool arguments, customer identifiers, and sensitive observations can create privacy and telemetry-cardinality risk.
• Cross-tenant and cross-profile aggregation must fail closed unless explicitly compatible.
• Missing evidence must remain visible as missing or indeterminate rather than being converted into success or failure.

Related: #43, #45, #46, #49, #53