Skip to content

Fix: don't adopt foreign global OTel parent for Judgment spans (regression from #749) - #752

Draft
claude[bot] wants to merge 1 commit into
mainfrom
hotfix/global-context-foreign-parent-adoption
Draft

Fix: don't adopt foreign global OTel parent for Judgment spans (regression from #749)#752
claude[bot] wants to merge 1 commit into
mainfrom
hotfix/global-context-foreign-parent-adoption

Conversation

@claude

@claude claude Bot commented Jul 6, 2026

Copy link
Copy Markdown

Requested by Konark · Slack thread

📝 Summary

  • 1. Decouple the READ and WRITE directions of OTel context propagation that Fix: Install as global tracer provider controls otel context  #749 conflated, so Judgment no longer adopts a foreign ambient parent from the global context by default.
  • 2. get_current_context() (READ) now returns Judgment's private runtime context by default → Judgment mints independent root traces again.
  • 3. Preserve Fix: Install as global tracer provider controls otel context  #749's intended feature (WRITE): Judgment's active span is still mirrored into the global context so external instrumentation nests under Judgment.
  • 4. Make ambient-global-parent adoption strictly opt-in via _use_global_context (now defaults to False even after global install). The customer's _use_global_context = False mitigation is now the default and stays valid.
  • 5. Add regression tests for a Judgment span created under an unsampled ambient foreign parent.

Before / After (plain language)

Before: When judgeval is installed as the global OpenTelemetry provider inside a long-lived server that already has ambient OTel spans (Google ADK on Vertex Agent Engine), Judgment silently stops recording its own traces. Every Judgment span gets re-parented under the ambient foreign request span (orphaned — Judgment's backend never sees that root), or is dropped by the default ParentBased sampler when that ambient parent isn't sampled. A short standalone script works only because its global context is empty at span-creation time.

After: Judgment creates its own independent root traces again by default. The outward-nesting behavior from #749 (external libraries nesting under Judgment's spans) is preserved. Adopting the ambient global parent is now explicit opt-in.

How

The bug was that #749 tied two opposite directions of propagation to a single _use_global_context flag that flipped to True on install_as_global_tracer_provider():

  • WRITE direction (keep — the intended feature): publish Judgment's active span into the global OTel context so external instrumentation nests under Judgment.
  • READ direction (the bug): read the global OTel context to pick the parent for Judgment's own spans.

Changes in src/judgeval/trace/judgment_tracer_provider.py:

  • New flag _mirror_active_span_to_global (WRITE). install_as_global_tracer_provider() now sets this flag (not _use_global_context).
  • _use_global_context (READ) now defaults to False and stays False after global install; it is the explicit opt-in knob to restore full Fix: Install as global tracer provider controls otel context  #749 read behavior (ambient-parent adoption).
  • get_current_context() returns the private _runtime_context unless _use_global_context is set.
  • attach_context() / detach_context() now dual-write: always attach to the private runtime context (so intra-Judgment nesting works when reading privately) and additionally attach to the global context when _should_write_to_global_context() is true (mirror on, or read-opt-in on). A small _DualContextToken tracks both tokens and unwinds them in reverse order.

Regression test

src/tests/trace/test_tracer_provider.pyTestForeignParentDecoupling:

  • test_span_under_unsampled_ambient_parent_still_exports_as_judgment_root: installs the global-provider WRITE state, attaches an unsampled foreign NonRecordingSpan (TraceFlags(0)) into the global OTel context, creates a Judgment span, and asserts it (a) is recording/sampled, (b) is exported via an in-memory collecting exporter, and (c) is an independent Judgment root — its trace_id differs from the foreign parent's and its parent is None.
  • test_intra_judgment_nesting_preserved: a Judgment child correctly parents under a Judgment parent (same trace, correct parent span id).
  • test_opt_in_use_global_context_restores_adoption: with _use_global_context = True, Judgment reads the global context and does adopt the ambient parent (full Fix: Install as global tracer provider controls otel context  #749 read behavior).
  • Updated TestGlobalContextBridge to reflect the WRITE/READ split (install enables WRITE, not READ).

Root cause

#749 made install_as_global_tracer_provider() set _use_global_context = True, which caused get_current_context() to read the global OTel context and adopt whatever ambient (foreign, often unsampled) span was current as the parent of Judgment's spans.

Open question for @abhi (sampler)

The sampler was intentionally left unchanged in this PR. My analysis suggested defaulting the inner TracerProvider's sampler to ALWAYS_ON as defense-in-depth (src/judgeval/trace/tracer.py ~line 227, currently sampler=sampler → SDK default ParentBased(root=ALWAYS_ON)), but this is deferred to you.

Empirically verified via the regression test: with the decoupling fix alone, the unsampled-ambient-parent test passes — because Judgment spans become independent roots, and the default ParentBased(root=ALWAYS_ON) samples roots regardless of the ambient parent's (unsampled) flags. Forcing ALWAYS_ON on the inner provider would be belt-and-suspenders and is a separable follow-up for you to decide. No sampler change was needed to make the regression test green.

Interim customer mitigation

Signal Advisors is applying JudgmentTracerProvider.get_instance()._use_global_context = False. This PR makes that value the default, so the mitigation becomes a no-op (still valid and harmless).

Note: this replaces the earlier PR #751 (same commits), which targeted main from a non-hotfix/ branch and tripped the validate-branch check. Re-opened from hotfix/global-context-foreign-parent-adoption to satisfy that rule.

✅ Checklist

🤖 Generated with Claude Code

https://claude.ai/code/session_01KKT33Etr45m6SfJRsx3uUX


Generated by Claude Code

Decouple the two directions of context propagation that #749 conflated.

READ direction: get_current_context() now returns Judgment's private
runtime context by default, so Judgment mints independent root traces and
no longer adopts a foreign ambient parent from the global OTel context
(e.g. Google ADK / Vertex Agent Engine request spans). Adopting the
ambient global parent is now strictly opt-in via _use_global_context,
which defaults to False even after install_as_global_tracer_provider().

WRITE direction: preserved. Judgment's active span is still mirrored into
the global context (new _mirror_active_span_to_global flag, enabled on
global install) so external instrumentation keeps nesting under Judgment.
attach_context/detach_context now dual-write to both the private runtime
context and the global context so intra-Judgment nesting is preserved
when reading from the private context.

The customer interim mitigation (_use_global_context = False) is now the
default and remains valid.

Adds regression tests: a Judgment span created under an unsampled ambient
foreign parent still records, exports, and is an independent Judgment
root; intra-Judgment nesting is preserved; and the opt-in still restores
adoption.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KKT33Etr45m6SfJRsx3uUX
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant