Categorical evaluation fails for A2A sessions — transcript builder missing A2A_INTERACTION content extraction

[evekhm]

Description

When using evaluate_categorical() on sessions that involve remote A2A agents (via ADK's TransferToAgentTool with RemoteA2aAgent), all such sessions receive parse_error=True with raw_response=None because AI.GENERATE returns NULL.

Root cause

The transcript-building SQL in categorical_evaluator.py uses this COALESCE to extract text from event content:

COALESCE(
  JSON_VALUE(content, '$.text_summary'),
  JSON_VALUE(content, '$.response'),
  JSON_VALUE(content, '$.tool'),
  ''
)

However, A2A_INTERACTION events (introduced in google/adk-python#5325) store the remote agent's response under a different JSON path:

{
  "artifacts": [{"parts": [{"kind": "text", "text": "The actual agent response..."}]}],
  "contextId": "...",
  "history": [...]
}

None of the existing JSON paths ($.text_summary, $.response, $.tool) match this structure. The transcript for A2A sessions looks like:

USER_MESSAGE_RECEIVED [knowledge_supervisor]: How many PTO days...
LLM_RESPONSE [knowledge_supervisor]: call: transfer_to_agent
TOOL_STARTING [knowledge_supervisor]: transfer_to_agent
A2A_INTERACTION [knowledge_supervisor]:              <-- EMPTY, 3000+ bytes of content but nothing extracted
AGENT_COMPLETED [pto_agent]:

The evaluation model sees a question with no answer, cannot classify, and returns NULL.

Impact

• 100% of A2A sessions get parse_error=True and category=None
• In our testing: 38-42 out of 100 sessions were A2A, so ~40% of all evaluations failed
• These show up as "UNKNOWN" in category distributions, inflating parse error rate

Reproduction

from bigquery_agent_analytics import Client, CategoricalEvaluationConfig, TraceFilter

client = Client(project_id=..., dataset_id=..., table_id=...)
# Configure metrics...
report = client.evaluate_categorical(config=cat_config, filters=TraceFilter(limit=100))

# Check parse errors - all A2A sessions will have raw_response=None
for sr in report.session_results:
    for mr in sr.metrics:
        if mr.parse_error and mr.raw_response is None:
            print(f"NULL result: {sr.session_id}")

Proposed fix

Add JSON_VALUE(content, '$.artifacts[0].parts[0].text') to the COALESCE chain in all 4 transcript-building locations in categorical_evaluator.py:

COALESCE(
  JSON_VALUE(content, '$.text_summary'),
  JSON_VALUE(content, '$.response'),
  JSON_VALUE(content, '$.artifacts[0].parts[0].text'),   -- NEW: A2A_INTERACTION events
  JSON_VALUE(content, '$.tool'),
  ''
)

Affected locations in categorical_evaluator.py:

1. CATEGORICAL_TRANSCRIPT_QUERY (line ~230)
2. CATEGORICAL_AI_GENERATE_QUERY (line ~256)
3. build_ai_classify_query() (line ~376)
4. build_ai_generate_query() (line ~445)

[caohy1988]

A couple of tightening suggestions after comparing this issue with latest main in both the SDK and the ADK BigQuery AA plugin:

1. The categorical evaluator diagnosis looks correct, but the issue is probably under-scoped. The same missing A2A extraction pattern also exists in insights.py, and the shared response extractor in event_semantics.py / udf_kernels.py still only understands flat keys like response, text_summary, text, and raw. If the intent is to fix only categorical evaluation, I’d say that explicitly; otherwise this reads more like a shared A2A response-text extraction gap.

2. It would help to include one sanitized sample content payload from a real A2A_INTERACTION row. In the plugin, A2A_INTERACTION logs raw_content = a2a:response, so the SDK is consuming the remote response payload directly rather than a normalized summary field. That supports the issue, but the proposed $.artifacts[0].parts[0].text path still looks like an observed shape rather than a guaranteed contract. A concrete sample would remove ambiguity.

3. I’d add acceptance criteria for regression tests in the shared extraction layer, not just the four SQL call sites. Right now the tests only cover flat-key extraction (response, text_summary, text, raw). Adding an A2A-shaped fixture to test_event_semantics.py / test_udf_kernels.py, plus a categorical evaluator regression test, would make the fix much safer.

[caohy1988]

Follow-up: code-level verification against main

I've walked the actual codebase to verify the issue and my earlier comment. The diagnosis is correct, but the scope is significantly broader than the 4 call sites identified.

Verification: the proposed fix targets are real

All four COALESCE chains in categorical_evaluator.py (lines 230–233, 255–258, 375–378, 443–446) use the same pattern:

COALESCE(
  JSON_VALUE(content, '$.text_summary'),
  JSON_VALUE(content, '$.response'),
  JSON_VALUE(content, '$.tool'),
  ''
)

None of these reach $.artifacts[0].parts[0].text. The proposed fix of adding that path to the COALESCE chain is correct for these four sites.

The gap is much wider than 4 sites

The same missing-A2A extraction pattern appears in 129 content extraction sites across 14 files:

File	Extraction sites
trace.py	21
views.py	16
context_graph.py	16
categorical_evaluator.py	12
trace_evaluator.py	11
insights.py	11
ai_ml_integration.py	8
feedback.py	7
ontology_graph.py	6
memory_service.py	6
udf_sql_templates.py	4
udf_kernels.py	4
evaluators.py	4
bigframes_evaluator.py	3

Every one of these will silently produce empty text for A2A_INTERACTION events.

A2A_INTERACTION is unknown to the event model

event_semantics.py defines EVENT_FAMILIES and ALL_KNOWN_EVENT_TYPES — neither includes A2A_INTERACTION. The RESPONSE_EVENT_TYPES tuple is ("LLM_RESPONSE", "AGENT_COMPLETED") only. Any feature that filters by event family or response-carrying events will skip A2A entirely.

The shared Python extractor doesn't handle nested content

udf_kernels.py:extract_response_text_from_dict (line 84) checks response, text_summary, text, raw — all flat top-level keys:

return (
    content.get("response")
    or content.get("text_summary")
    or content.get("text")
    or content.get("raw")
    or None
)

This is the shared core used by event_semantics.extract_response_text, so every Python-side extraction path also misses A2A content.

Recommended approach

Rather than patching 129 sites individually, the fix should be layered:

1. Add A2A_INTERACTION to event_semantics.py — new entry in EVENT_FAMILIES (e.g., "a2a": ["A2A_INTERACTION"]) and ensure it's in ALL_KNOWN_EVENT_TYPES.

2. Fix the shared Python extractor — add A2A content traversal to extract_response_text_from_dict in udf_kernels.py:

   # A2A_INTERACTION: nested artifact response
   artifacts = content.get("artifacts")
   if isinstance(artifacts, list) and artifacts:
       parts = artifacts[0].get("parts", [])
       if isinstance(parts, list) and parts:
           return parts[0].get("text")

3. Create a shared SQL snippet for the JSON_VALUE COALESCE chain (a module-level constant in event_semantics.py or a builder function), then replace the 129 inline COALESCE chains with a reference to it. This is the heavier lift but prevents the problem from recurring every time a new content shape appears.

4. As a quick win, patch the 4 categorical_evaluator sites as proposed in the issue — this unblocks the reporter's immediate use case while the broader fix is planned.

Scoping suggestion: this issue could stay focused on categorical evaluation (the quick win) and spawn a follow-up for the shared-extractor + event-model work.