Skip to content

product: calculate and surface per-session evidence health #240

Description

@Siddhant-K-code

Goal

Tell a reviewer whether a session record is complete enough for the decision at hand, and explain why it may be partial or ambiguous.

This implements the evidence-trust requirement from the Verio telemetry learning: a clean timeline must not be presented as proof of complete capture.

Problem

The current UI can show a running or completed session and raw events, but it does not calculate a first-class evidence-health result. Missing boundaries, unpaired events, provider limitations, stale metadata, or export failures can be easy to miss.

A generic percentage would be misleading because AgentTrace cannot count events a provider never exposes. Health must combine observed structural checks with the declared limits of the capture method.

Scope

Define a deterministic, versioned evidence-health result with:

  • status such as healthy, partial, unknown, or invalid;
  • observed session start and end boundaries;
  • capture provider, method, and applicable capture-matrix version;
  • unpaired tool calls/results and other broken event relationships;
  • timestamp regressions or suspicious discontinuities;
  • stale or unfinalized session metadata;
  • recorded export failures or dropped-event signals when observable;
  • explicit provider blind spots; and
  • machine-readable reason codes plus human explanations.

Expose the same result in:

  • a CLI inspection path;
  • replay/session summary output;
  • JSON or MCP/API session metadata; and
  • the local web dashboard session header.

A status must never imply that unobservable provider activity was complete.

Design constraints

  • Keep the calculation local and dependency-free.
  • Separate observed defects from declared provider limitations.
  • Preserve source provenance for every reason.
  • Do not infer misconduct, policy failure, or compliance from missing evidence.
  • Make the rules testable against complete, deliberately partial, legacy, and malformed fixtures.

Acceptance criteria

  • The health schema and reason codes are documented and versioned.
  • Missing start/end boundaries, unpaired tool events, timestamp regressions, and stale metadata are detected.
  • Provider blind spots are sourced from the capture matrix rather than guessed.
  • Complete, partial, legacy, active, and malformed session fixtures have deterministic results.
  • CLI, JSON/API, replay, and local dashboard show consistent status and reasons.
  • No health state claims capture completeness beyond what the provider exposes.
  • Export/drop failures are shown when AgentTrace has observable evidence of them.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions