v0.7.1

v0.7.1

A hotfix for the Foundry operations dashboard shipped in v0.7.0. If you tried agentops telemetry dashboard deploy on v0.7.0 and it failed, upgrade to v0.7.1 and it works.

Fixed

• agentops telemetry dashboard deploy now deploys. In v0.7.0 the generated ARM template set the workbook location to global, which Microsoft.Insights/workbooks does not accept. Every deploy failed with LocationNotAvailableForResourceType. The workbook now deploys to the target resource group's region by default ([resourceGroup().location]), and an explicit region override is still honored. Verified with a live deploy against Azure: the workbook resource is created and reports the resource group region.

Security

• Bumped idna to 3.18 and pytest to 9.1.1 in uv.lock to clear two moderate Dependabot alerts.
• The high-severity nltk advisory (GHSA-p4gq-832x-fm9v) has no upstream fix yet; the latest published nltk is still the affected version, so that alert stays open until a patched release lands.

Upgrade

pip install --upgrade agentops-accelerator==0.7.1

No configuration or workflow changes are required. If you already exported or imported the workbook JSON manually as a v0.7.0 workaround, no action is needed.

v0.7.0

v0.7.0

This release adds a Foundry operations dashboard for Azure OpenAI, a new doctor check for usage telemetry, and a redesigned Cockpit that answers "can I ship?" first.

Tracked in #343 and delivered in #344.

Highlights

• Foundry operations dashboard. A new Azure Monitor Workbook, deployable from the CLI, gives you four views of a single Azure OpenAI resource against a Log Analytics workspace: capacity (PTU utilization, rate-limit, spillover), traffic and tokens, latency percentiles (TTFT, TBT, TTLT, tokens/sec), and errors and throttling. The workbook JSON, its per-metric KQL queries, and an authoring README ship in the package.
• Doctor now flags missing AOAI usage telemetry. A new WAF-AI Operational Excellence rule (waf.observability.aoai_diagnostic_categories) warns when your Azure OpenAI account is not emitting the RequestResponse and AzureOpenAIRequestUsage diagnostic categories to a Log Analytics workspace, and prints the exact az monitor diagnostic-settings create command to fix it. Doctor stays read-only.
• Cockpit redesign. Cockpit now opens with three consolidated status cards (Readiness, Doctor, Eval gate) that expand on click, promotes Next actions to second position, and collapses detail sections by default. The former Eval gate summary and Quality gate summary are merged into a single Eval gates section, a Foundry operations dashboard tile is added to the launchpad footer, and a duplicated App Insights CTA is removed. Cockpit remains read-only.
• Docs. New Operate pages for the Foundry operations workbook and the KQL library, plus recipe, related-skills, and next-steps sections across the Evaluate, Ship, Observe, and Operate pages. The Overview is rewritten as a funnel.

New: the first CLI command that creates an Azure resource

Everything in agentops has been read-only until now. agentops telemetry dashboard deploy is the first command that creates an Azure resource. It is scoped to a single Microsoft.Insights/workbooks resource and nothing else. It runs an RBAC and diagnostic-settings preflight before deploying, and --dry-run emits the ARM template without deploying.

How to use it

agentops telemetry dashboard deploy    # preflight, then deploy the workbook (use --dry-run to see the ARM template)
agentops telemetry dashboard open      # open the workbook in the Azure Portal (use --print-url for non-interactive shells)
agentops telemetry dashboard export    # copy the packaged workbook JSON to a local path

Prerequisites

• RBAC for deploy: Workbook Contributor on the target resource group, and Log Analytics Reader on the workspace.
• Diagnostic categories: the AzureDiagnostics-based tiles require the RequestResponse and AzureOpenAIRequestUsage categories to be flowing from your Azure OpenAI account to the workspace. Run agentops doctor to check this.

Known limitation

The workbook JSON was authored from the issue's KQL spec and the Microsoft.Insights/workbooks schema without a live Azure environment. Smoke-test the four tiles against a real Log Analytics workspace before relying on them. Tiles that read from AzureDiagnostics will stay empty until the two diagnostic categories above are flowing.

v0.6.0

What's Changed

• fix(tests): install azure-mgmt-authorization in dev to unblock Release by @placerda in #311
• fix(eval): support azd ai agent eval generate with init fallback by @placerda in #320
• fix: declare click as an explicit runtime dependency by @Dongbumlee in #267
• chore(deps): bump actions/checkout from 6 to 7 by @dependabot[bot] in #329
• docs: clarify response context capture by @koriyoshi2041 in #334
• Use sandbox for prompt-agent PR candidates by @placerda in #325
• chore(deps): bump pydantic-settings from 2.14.1 to 2.14.2 by @dependabot[bot] in #333
• chore(deps): bump pyjwt from 2.12.1 to 2.13.0 by @dependabot[bot] in #313
• chore(deps): refresh Python dependency lock by @placerda in #335
• Add retrieval telemetry imports for evaluations by @placerda in #341
• Release v0.6.0 by @placerda in #342

New Contributors

• @koriyoshi2041 made their first contribution in #334

Full Changelog: v0.5.3...v0.6.0

v0.5.3

Full Changelog: v0.5.2...v0.5.3

v0.5.2

What's new

Per-evaluator input_mapping for grey-box RAG evaluation.

Evaluator overrides in agentops.yaml now accept an optional input_mapping map that is merged onto the preset's default inputs. You only list the keys you want to change, and the rest of the preset is preserved.

This is the piece that connects the multi-field HTTP response capture shipped in v0.5.0 (response_fields + \.<name>) to the RAG evaluators. You can now point a RAG evaluator at the live retrieved context captured from a JSON HTTP target, instead of static dataset context:

response_fields:
  context: context
evaluators:
  - name: GroundednessEvaluator
    input_mapping:
      context: \.context
  - name: RetrievalEvaluator
    input_mapping:
      context: \.context

The mapping applies to both explicitly listed overrides and auto-selected presets, and never mutates the shared evaluator catalog. A bare evaluator-name string (- GroundednessEvaluator) is still accepted as shorthand for { name: GroundednessEvaluator }, so existing configs keep working unchanged.

Install

pip install agentops-accelerator==0.5.2

Validation

• ruff check clean.
• Full unit suite: 1030 passed, 1 skipped (release build gate).

Full Changelog: v0.5.1...v0.5.2

v0.5.1

What's new

Rendered gate results in the GitHub Actions job summary. When AgentOps runs inside GitHub Actions, the gates now write their results straight to the workflow run summary, so reviewers read the report on the run page without downloading artifacts.

• agentops eval run appends the full rendered report.md to the run summary.
• agentops assert run and agentops redteam run append a concise pass/fail summary (suite, cases, pass rate, plus per-dimension and per-risk-category breakdowns).
• Writes are best-effort and a no-op outside GitHub Actions, so local runs are unaffected.

Generated workflows use Node24-ready action versions. The prompt-agent and watchdog workflow templates now pin actions/download-artifact@v7 instead of the Node20 @v4, so freshly generated pipelines no longer emit the "Node.js 20 actions are deprecated" warning. A regression guard checks every workflow template against the known Node20 action majors.

Upgrade notes

• The rendered-report feature is automatic in CI. No config change is required. Pipelines that install AgentOps from @main or upgrade to v0.5.1 pick it up on the next run.
• If you generated workflows with an earlier version, re-run agentops workflow generate (or bump the action majors by hand) to clear any remaining Node20 deprecation warnings.

Links

• Render gate results in Actions summary + Node24 workflow templates: #331
• Grey-box multi-field JSON capture (shipped in v0.5.0): #330

Validation

• Release build job: ruff + full unit test suite passed.
• Published to PyPI and verified on TestPyPI.

v0.5.0

What changed

This release lets AgentOps score the retrieval an agent actually used at eval time, instead of only the static context stored in your dataset. It is the engine piece that unlocks RAG evaluators (Groundedness, Retrieval, Document Retrieval) against a live HTTP orchestrator.

Added

• Grey-box retrieval capture for HTTP JSON targets. An HTTP target can now capture extra named fields from a JSON response through a response_fields map (name -> dot-path). Captured values are exposed to evaluator input_mapping as $response.<name> (for example $response.context, $response.retrieved_documents), and dataset columns can be referenced with $row.<name> (for example $row.qrels).
• This lets RAG evaluators such as Groundedness, Retrieval, and Document Retrieval score the retrieval that was actually used during the run, rather than static dataset context.

Compatibility

• Fully backward compatible. The primary prediction (response_field) and existing single-field behavior are unchanged when response_fields is not set. Multi-field capture applies to JSON-mode HTTP targets only.

Docs

• HTTP target reference now documents response_fields with a grey-box example, and the bundle input-mapping reference documents the $response.<name> and $row.<name> tokens.

Install

pip install agentops-accelerator==0.5.0

Validation

• Release pipeline build gate (ruff + full unit suite + package) passed; published to TestPyPI and PyPI.

v0.4.5

What this release is about

This release lets AgentOps run its safety and quality gates (ASSERT and Red Team) against a live HTTP orchestrator endpoint, not only model or deployment targets. With this, an HTTP agent such as the GPT-RAG orchestrator can be governed end to end in CI: evaluation, ASSERT policy checks, and Red Team, with the generated GitHub Actions and Azure DevOps pipelines running those gates automatically.

It also fixes a reasoning-model judge error that could fail the eval gate in CI.

Merged via #326 (feature), #327 (release lint hotfix), and #328 (release test hotfix).

Added

• Governance gates for HTTP agents (ASSERT and Red Team). agentops assert run and agentops redteam run now work against a live HTTP orchestrator endpoint. Red Team wraps the HTTP endpoint as an SDK-compatible target and reuses the AgentOps HTTP mapping (request_field, response_mode, stream, custom headers). ASSERT resolves assert-ai inside the active virtual environment, accepts non-secret values from assert.env, can request an AAD token from the Azure CLI for local auth-disabled Azure AI resources, injects the GPT-5 max_completion_tokens shim only when configured, and materializes a runtime ASSERT config so committed configs no longer need absolute artifact paths.
• Generated workflows run the ASSERT and Red Team gates. agentops workflow generate now installs the optional ASSERT/Red Team dependencies, runs those gates when assert: or redteam: is present in agentops.yaml, uploads their artifacts, and emits the corrected Red Team command quoting.

Fixed

• Reasoning-model judges no longer fail the eval gate in CI. The generated GitHub Actions and Azure DevOps eval and Red Team steps now forward AZURE_OPENAI_MODEL_NAME, so AgentOps detects reasoning models (such as gpt-5-nano) and uses max_completion_tokens instead of max_tokens. This removes the judge 400 error that could break the eval gate when the judge deployment is a reasoning model.

Upgrade notes

The CLI install spec in generated workflows can now target @main again. The HTTP governance code is on main as of this release, so projects that pinned a temporary feature branch can drop the pin. The HTTP agent tutorial no longer documents a temporary install pin.

Validated

• A live GPT-RAG orchestrator project ran both generated pipelines green: the PR workflow (sandbox deploy candidate plus eval, ASSERT, and Red Team gates) and the dev deploy workflow (eval gate plus azd deploy), using OIDC and azd deploy only.
• Unit suites for the generator and the ASSERT/Red Team runners pass.

v0.4.4

Evaluate streaming agents directly, no adapter

AgentOps http_json targets can now evaluate agents that stream their answers
(Server-Sent Events or raw text), not just agents that return a single JSON
body. This means you can point an eval straight at a streaming endpoint such as
the gpt-rag-orchestrator /orchestrator route without writing and hosting a
thin non-streaming adapter first.

Defaults are unchanged: existing http_json targets keep parsing a single JSON
response exactly as before (response_mode: json is the default and is
byte-for-byte compatible).

What you can do now

• Set response_mode: sse or response_mode: text on an http_json target to
  read a streamed body and aggregate it into one answer for scoring.
• Configure aggregation with an optional stream block:
  • text_field: dotted path to the token text when each SSE data: line is
    JSON (for example choices.0.delta.content).
  • done_marker: stop token, for example [DONE].
  • strip_leading_token: drop the first whitespace-delimited token. The
    gpt-rag orchestrator emits the conversation_id as the first chunk, so this
    removes it from the scored answer.
• Use a non-bearer auth header for endpoints gated by a shared secret. Two new
  fields, auth_header_name (default Authorization) and
  auth_value_template (default Bearer {token}), let you send something like
  X-API-KEY: <secret> without putting the secret in agentops.yaml. The
  {token} placeholder is replaced by the value of the auth_header_env
  environment variable at run time.

When a JSON parse fails on a text/event-stream response, the error now tells
you to set response_mode: sse or response_mode: text, so the misconfiguration
is obvious instead of a raw decode error.

Example: gpt-rag-orchestrator

target:
  kind: http_json
  url: https://<your-orchestrator>/orchestrator
  request_field: ask
  response_mode: text
  auth_header_name: X-API-KEY
  auth_value_template: "{token}"
  auth_header_env: ORCHESTRATOR_APP_APIKEY
  stream:
    strip_leading_token: true

Compatibility and transport

• No change for existing targets. response_mode defaults to json.
• Streaming reuses the same Python standard-library (urllib) transport and the
  same 3-try backoff as the JSON path. The body is read to completion (eval
  answers are bounded) and then aggregated, so retries stay robust.

Validation

• Full unit suite: 991 passed, 1 skipped (14 net-new tests, no regressions).
• New tests cover SSE/text aggregation, the leading-token strip, done_marker
  and event: error handling, the configurable auth header, and the
  response_mode validation gate.

New config fields

Field	Default	Purpose
response_mode	json	Selects the response parser: json, sse, or text.
stream.text_field	none	Dotted path to token text inside each JSON SSE data: line.
stream.done_marker	none	Token that ends the stream, e.g. [DONE].
stream.strip_leading_token	false	Drops the first whitespace-delimited token (e.g. a conversation_id prefix).
auth_header_name	Authorization	Header used to send the auth value.
auth_value_template	Bearer {token}	Template for the auth value; {token} is replaced by auth_header_env.

Full changelog: see CHANGELOG.md ([0.4.4]).

v0.4.3

[0.4.3] - 2026-06-17

Added

• Prompt-agent tutorials no longer require manual portal copy/paste.
  agentops prompt pull reads the configured Foundry prompt agent
  (agent: name:version), validates that the Foundry definition is actually a
  prompt agent, and writes the reviewed Sandbox instructions to
  .agentops/prompts/<agent-name>.prompt.md by default. Before writing, the CLI
  prints the resolved agent, endpoint, endpoint source, and destination file so
  operators can catch the wrong environment early. Changed prompt files are
  protected by default and require --force to overwrite reviewed local edits.
  The command updates prompt_file in agentops.yaml unless
  --no-update-config is passed, and it can resolve the endpoint from
  --project-endpoint, agentops.yaml, AZURE_AI_FOUNDRY_PROJECT_ENDPOINT, or
  the active .azure/<env>/.env. The prompt-agent tutorial and packaged
  agentops-eval skill now use this command instead of a manual here-string.
  (#322)

Changed

• agentops eval init now recommends evaluators from the agent and dataset
  shape. The azd bootstrap path now reuses the same AgentOps evaluator
  catalog as agentops eval run: free-form answer datasets get answer-quality
  checks, RAG-shaped datasets get groundedness / relevance / retrieval checks,
  and tool-use datasets get tool-call / intent / task-adherence checks while
  avoiding literal-answer similarity metrics. Explicit evaluators: entries in
  agentops.yaml still win. The CLI prints the recommendation source, detected
  signals, and selected azd built-ins before reporting the generated
  eval.yaml, so users can see why those evaluators were chosen.
  (#323)