fix(ci): add langchain-core to [dev] so test collection passes - #4
Merged
Conversation
The SDK's import chain (nullrun.__init__ -> nullrun.decorators ->
nullrun.instrumentation.langgraph -> 'from langchain_core.callbacks
import BaseCallbackHandler') runs at pytest *collection* time, not at a
specific test. With CI installing [dev] only, every test in the suite
errored on collection with:
ModuleNotFoundError: No module named 'langchain_core'
This is the same class of bug that 'nullrun[langgraph]' exists to
prevent for end users, except the dev install never benefited from
the extras indirection.
Fix: add 'langchain-core>=0.3,<1.0' to the [dev] extras. The
heavier 'langgraph' / 'langchain' extras pull in stacks the unit
tests don't use; the bare core is the smallest dep that makes the
import chain resolve and unblocks test collection on every
supported Python (3.10 / 3.11 / 3.12) on every PR.
Validation: locally on Python 3.14.2 (which is outside the
3.10/3.11/3.12 matrix that CI tests), 'pip install -e .[dev]'
followed by 'pytest tests/' runs 443/443 + 9/9 new byte-mismatch
unit tests, no collection error. CI will re-confirm on the 3.10 /
3.11 / 3.12 matrix.
maltsev-dev
added a commit
that referenced
this pull request
Jul 4, 2026
Bug-fix release layered on top of 0.12.1. No wire-format change; both fixes are client-side only. * BUG #4 -- check_workflow_budget() now sends a fresh uuidv7 as the "execution_id" field on every /check call instead of reusing workflow_id. The server's gate_reserve_v3 overwrites the field on response anyway, but a client-side placeholder that collides across calls confuses the v3 reservation binding on /track when Transport.track_single() reaches the backend and the field is stale -- exact symptom is 503 RESERVATION_NOT_FOUND per CLAUDE.md section 29. * BUG #5 -- new nullrun.runtime._GATE_CACHE (5s TTL, keyed on (workflow_id, chain_id, model)) collapses consecutive /gate calls from inside `with chain(...)` to a single roundtrip, avoiding 100 /gate calls per 100-step agent loop. Single-shot (Hard mode) callers MUST bypass the cache -- Hard mode's binary allow -> block semantics would let a stale "allow" leak a budget-exhausted call through. Opt-out via NULLRUN_GATE_CACHE_DISABLE=1 for callers that want the legacy always-roundtrip behaviour (used by live smoke tests per docs/runbooks/budget-blue-green-smoke.sh). Tests: 158 new lines in tests/test_v3_wire_contract.py covering per-call execution_id uniqueness, uuidv7 format validation, and the new cache data-structure invariants + opt-out cases. Bumps __version__ + pyproject.toml to 0.12.2.
maltsev-dev
added a commit
that referenced
this pull request
Jul 4, 2026
`ruff check src/` flagged that the BUG #4 line `from nullrun.uuid7 import uuid7_str # CLAUDE.md §24` landed mid-way through the first-party import block (between `nullrun.context` and `nullrun.observability`), breaking I001 import sort. Moved to the bottom of the first-party block (alphabetic order — `uuid7` sorts after `transport`). Also lets ruff auto-fix two cosmetic cleanups in tests/test_v3_wire_contract.py: * sort `_V3_ERROR_CODE_MAP` alphabetic in the existing transport import group (was below `_parse_v3_error_envelope`); * drop a stray blank-line gap between two top-level `from nullrun.transport import (...)` groups. `ruff check src/ tests/` after the fix: 8 pre-existing I001 findings remain in unrelated test files (predicate, test_circuit_breaker_branches.py, test_framework_patches.py, etc.) — out of scope for this PR. Scope above matches the CI step (`ruff check src/`). No behavioural change.
maltsev-dev
added a commit
that referenced
this pull request
Jul 4, 2026
* release(0.12.2): fresh execution_id per /check + chain-mode /gate cache Bug-fix release layered on top of 0.12.1. No wire-format change; both fixes are client-side only. * BUG #4 -- check_workflow_budget() now sends a fresh uuidv7 as the "execution_id" field on every /check call instead of reusing workflow_id. The server's gate_reserve_v3 overwrites the field on response anyway, but a client-side placeholder that collides across calls confuses the v3 reservation binding on /track when Transport.track_single() reaches the backend and the field is stale -- exact symptom is 503 RESERVATION_NOT_FOUND per CLAUDE.md section 29. * BUG #5 -- new nullrun.runtime._GATE_CACHE (5s TTL, keyed on (workflow_id, chain_id, model)) collapses consecutive /gate calls from inside `with chain(...)` to a single roundtrip, avoiding 100 /gate calls per 100-step agent loop. Single-shot (Hard mode) callers MUST bypass the cache -- Hard mode's binary allow -> block semantics would let a stale "allow" leak a budget-exhausted call through. Opt-out via NULLRUN_GATE_CACHE_DISABLE=1 for callers that want the legacy always-roundtrip behaviour (used by live smoke tests per docs/runbooks/budget-blue-green-smoke.sh). Tests: 158 new lines in tests/test_v3_wire_contract.py covering per-call execution_id uniqueness, uuidv7 format validation, and the new cache data-structure invariants + opt-out cases. Bumps __version__ + pyproject.toml to 0.12.2. * fix(lint): reorder uuid7 import in runtime.py per ruff I001 `ruff check src/` flagged that the BUG #4 line `from nullrun.uuid7 import uuid7_str # CLAUDE.md §24` landed mid-way through the first-party import block (between `nullrun.context` and `nullrun.observability`), breaking I001 import sort. Moved to the bottom of the first-party block (alphabetic order — `uuid7` sorts after `transport`). Also lets ruff auto-fix two cosmetic cleanups in tests/test_v3_wire_contract.py: * sort `_V3_ERROR_CODE_MAP` alphabetic in the existing transport import group (was below `_parse_v3_error_envelope`); * drop a stray blank-line gap between two top-level `from nullrun.transport import (...)` groups. `ruff check src/ tests/` after the fix: 8 pre-existing I001 findings remain in unrelated test files (predicate, test_circuit_breaker_branches.py, test_framework_patches.py, etc.) — out of scope for this PR. Scope above matches the CI step (`ruff check src/`). No behavioural change.
maltsev-dev
added a commit
that referenced
this pull request
Aug 7, 2026
The SDK's import chain (nullrun.__init__ -> nullrun.decorators ->
nullrun.instrumentation.langgraph -> 'from langchain_core.callbacks
import BaseCallbackHandler') runs at pytest *collection* time, not at a
specific test. With CI installing [dev] only, every test in the suite
errored on collection with:
ModuleNotFoundError: No module named 'langchain_core'
This is the same class of bug that 'nullrun[langgraph]' exists to
prevent for end users, except the dev install never benefited from
the extras indirection.
Fix: add 'langchain-core>=0.3,<1.0' to the [dev] extras. The
heavier 'langgraph' / 'langchain' extras pull in stacks the unit
tests don't use; the bare core is the smallest dep that makes the
import chain resolve and unblocks test collection on every
supported Python (3.10 / 3.11 / 3.12) on every PR.
Validation: locally on Python 3.14.2 (which is outside the
3.10/3.11/3.12 matrix that CI tests), 'pip install -e .[dev]'
followed by 'pytest tests/' runs 443/443 + 9/9 new byte-mismatch
unit tests, no collection error. CI will re-confirm on the 3.10 /
3.11 / 3.12 matrix.
maltsev-dev
added a commit
that referenced
this pull request
Aug 7, 2026
* release(0.12.2): fresh execution_id per /check + chain-mode /gate cache Bug-fix release layered on top of 0.12.1. No wire-format change; both fixes are client-side only. * BUG #4 -- check_workflow_budget() now sends a fresh uuidv7 as the "execution_id" field on every /check call instead of reusing workflow_id. The server's gate_reserve_v3 overwrites the field on response anyway, but a client-side placeholder that collides across calls confuses the v3 reservation binding on /track when Transport.track_single() reaches the backend and the field is stale -- exact symptom is 503 RESERVATION_NOT_FOUND per CLAUDE.md section 29. * BUG #5 -- new nullrun.runtime._GATE_CACHE (5s TTL, keyed on (workflow_id, chain_id, model)) collapses consecutive /gate calls from inside `with chain(...)` to a single roundtrip, avoiding 100 /gate calls per 100-step agent loop. Single-shot (Hard mode) callers MUST bypass the cache -- Hard mode's binary allow -> block semantics would let a stale "allow" leak a budget-exhausted call through. Opt-out via NULLRUN_GATE_CACHE_DISABLE=1 for callers that want the legacy always-roundtrip behaviour (used by live smoke tests per docs/runbooks/budget-blue-green-smoke.sh). Tests: 158 new lines in tests/test_v3_wire_contract.py covering per-call execution_id uniqueness, uuidv7 format validation, and the new cache data-structure invariants + opt-out cases. Bumps __version__ + pyproject.toml to 0.12.2. * fix(lint): reorder uuid7 import in runtime.py per ruff I001 `ruff check src/` flagged that the BUG #4 line `from nullrun.uuid7 import uuid7_str # CLAUDE.md §24` landed mid-way through the first-party import block (between `nullrun.context` and `nullrun.observability`), breaking I001 import sort. Moved to the bottom of the first-party block (alphabetic order — `uuid7` sorts after `transport`). Also lets ruff auto-fix two cosmetic cleanups in tests/test_v3_wire_contract.py: * sort `_V3_ERROR_CODE_MAP` alphabetic in the existing transport import group (was below `_parse_v3_error_envelope`); * drop a stray blank-line gap between two top-level `from nullrun.transport import (...)` groups. `ruff check src/ tests/` after the fix: 8 pre-existing I001 findings remain in unrelated test files (predicate, test_circuit_breaker_branches.py, test_framework_patches.py, etc.) — out of scope for this PR. Scope above matches the CI step (`ruff check src/`). No behavioural change.
maltsev-dev
added a commit
that referenced
this pull request
Aug 13, 2026
…#89) * fix(sdk): 5xx + invalid-JSON + compromised-wording + request_timeout (RUN_ID 20260811-1) Closes 4 SDK defects from NULLRUN QA cycle 20260811-1: * DEF-ERRHDL-AUTH-PATH-CODE-PIN-01 (Medium) -- _authenticate() in runtime.py now routes 5xx to NullRunBackendError (NR-B002) instead of NullRunAuthenticationError (NR-A001). 401 keeps NullRunAuthenticationError + NR-A003; other 4xx keep NR-A001. Pre-fix operators were nudged to rotate valid keys during backend outages ("API key may be invalid or expired" for status=500 is misleading). Per CLAUDE.md §13 5xx is a backend-class error, not auth-class. * DEF-ERRHDL-INVALID-JSON-01 (Medium) -- new _safe_json() helper in transport.py wraps json.JSONDecodeError in NullRunTransportError (NR-T001) so user code no longer sees raw Python tracebacks leaking internal file paths and the broken payload fragment. Body preview truncated to 200 chars to prevent log flooding + PII leak. The 200-OK path in runtime.py:_authenticate() now calls _safe_json instead of response.json(). * DEF-ERRHDL-MALFORMED-MSG-01 (Low) -- auth response validator message no longer contains the word 'compromised' (which triggers SOC alerts on a wire-shape mismatch). Replacement wording: 'server returned an unexpected response shape'. * DEF-ERRHDL-NO-TIMEOUT-01 (Medium) -- NullRunRuntime.__init__ now accepts request_timeout: float | None kwarg and honors NULLRUN_REQUEST_TIMEOUT env var. Precedence: kwarg > env > default(30). Malformed env falls back to 30 (don't crash init). The kwarg exposes the surface; full wire-up to httpx.Client.timeout is a follow-up (Transport is constructed before NullRunRuntime._timeout is set). Source-pin regression tests: tests/test_2026_08_11_fixes.py (6 tests) pin the fixes so future refactors cannot silently revert. Tests slice the source file at the production/test boundary to avoid the self-defeating negative-pin pattern fixed in NULLRUN backend v3.37 / commit 131699fd. Wire contract: additive. NR-T001 is a new code; existing NR-A*/NR-B* codes unchanged. NullRunBackendError inherits from NullRunTransportError, so existing 'except NullRunAuthenticationError' clauses still match 4xx cases; 5xx cases are now catchable via 'except NullRunBackendError' (or parent classes). NULLRUN defect log: docs/runbooks/2026-08-11-sdk-fixes.md in the NULLRUN repo (separate runbook, separate commit there). * test(sdk): fix 2 over-strict / stale source-pin tests from RUN_ID 20260811-1 Two regressions surfaced after rebase of 58b8aa6 onto origin/master (0.15.0). Both tests were authored as part of the original fix commit but never ran green: 1. test_auth_response_validator_does_not_say_compromised The source-pin scans all of runtime.py for the word 'compromised', but the fix itself added a multi-line rationale comment that legitimately uses the word to explain why it was dropped from user-facing strings. The pin needs to scan code (string literals) not comments. Add _strip_comment_lines() and apply it before the assertions. 2. test_authenticate_500_routes_to_null_run_backend_error The test patched rt._transport._client.post.return_value, which was the pre-0.15.0 contract. The 0.15.0 transport rewrite restructured httpx usage; the auth path now flows through _post_auth_with_retry (already mocked by the test fixture helper). Switch the patch target to rt._post_auth_with_retry.return_value. Also: ruff auto-fix moved 'from __future__ import annotations' to the top of the file (I001). Verified: 1502 passed, 7 skipped on full suite; ruff + mypy clean. * fix(sdk): H6 BUDGET_RECHECK_FAILED + L5 ACK HMAC + L6 wire audit + M8 capabilities shape (audit 2026-08-12 WIP) H6 — post-approval budget recheck gets a typed exception (NullRunBudgetRecheckFailedError, NR-B006) with current_spend_cents + budget_cents first-class attributes so callers can compute the remaining cap and decide whether to retry after re-/gate. Wire code BUDGET_RECHECK_FAILED mapped in _V3_ERROR_CODE_MAP. Pre-fix SDK 0.14.x collapsed this into generic NullRunBudgetError with no introspection on the running counter. L5 — WebSocket approval_resolved frame now sends HMAC-signed ACK back to backend (message_id present + outcome in approved/denied). Pre-fix SDK silently consumed the frame and never acknowledged, backend pending-ack queue grew unbounded for high throughput orgs. Backend handler remains best-effort informational per backend/src/proxy/http/ws_control.rs:842-848, but wire-up closes the missing-ACK gap. L6 — runtime.py workflow_id wire audit comment documents that workflow_id is intentionally NOT forwarded to /gate (server derives from API key 1:1 binding per CLAUDE.md §12); flows into /track + /events via _enrich_event for cost attribution. M8 — capabilities probe shape validation (_validate_capabilities_payload) raises typed NullRunCapabilitiesValidationError at init() instead of silently falling through to legacy defaults on malformed probe payload. * extend exceptions * test(sdk): BUDGET_RECHECK_FAILED dispatch to typed NR-B006 (audit H6 closure) * fix(sdk): flip Transport.execute fallback default to STRICT (audit #4) Pre-v3.53 ExecuteConfig.fallback_mode, Transport.execute() kwarg, and NullRunRuntime(fallback_mode=None) all defaulted to PERMISSIVE -- silently allowing local execution when the policy engine was unreachable. /api/v1/execute is the PRIMARY enforcement point (per transport.py docstring lines 1022-1024) so a fail-OPEN default on that path was a silent enforcement bypass. Per CLAUDE.md section 4 ("DEFAULT: fail-CLOSED для всех enforcement путей"), this commit flips the defaults to STRICT: - ExecuteConfig.fallback_mode: STRICT - Transport.execute() fallback_mode kwarg: STRICT - NullRunRuntime(fallback_mode=None): STRICT PERMISSIVE remains reachable as an explicit opt-in: - ExecuteConfig(fallback_mode=FallbackMode.PERMISSIVE) - Transport.execute(..., fallback_mode=FallbackMode.PERMISSIVE) - NullRunRuntime(..., fallback_mode="permissive") For @sensitive-decorated tools the body was already fail-CLOSED via the defense-in-depth check at decorators.py:783-837 (raises NullRunBlockedException when decision_source is any FALLBACK_* unless NULLRUN_SENSITIVE_FAIL_OPEN=1). That defense layer is unchanged. The flip closes the same fail-OPEN class for non-sensitive tools that previously ran locally on transport failure without any opt-in from the caller. Changes: - transport.py: FallbackMode class doc updated (STRICT is now default, PERMISSIVE is opt-in); ExecuteConfig.fallback_mode default = STRICT; Transport.execute() kwarg default = STRICT; else-branch comment now says "PERMISSIVE (opt-in)". - runtime.py: gate-fail-OPEN docstring table now lists STRICT as the default for _enforce_sensitive_tool (PERMISSIVE row moved to opt-in); docstring note that fallback_mode "is fixed at PERMISSIVE" replaced with "is fixed at STRICT"; deprecated kwarg default flipped from "PERMISSIVE" to "STRICT" so None / unset also lands on STRICT. - tests/test_transport.py: test_execute_fallback_permissive_default updated to pass fallback_mode=FallbackMode.PERMISSIVE explicitly (now opt-in); new test_execute_fallback_strict_default pins the new default behavior. - tests/test_transport_branches.py: same pair. - tests/test_no_local_policy.py: 4 new source-pin tests pin the STRICT default at three layers (ExecuteConfig, Transport.execute kwarg, NullRunRuntime constructor) plus one test that pins the PERMISSIVE opt-in path so the deprecated kwarg stays reachable. Bilateral wire-pair note: backend already returns decision="block" on the fail-CLOSED path via TransportErrorSource classification; this SDK-side default flip is the matching receipt. No backend changes required for audit #4. 1529 passed, 7 skipped (no regressions). * fix(sdk): MCPAdapter.call_tool routes through gate when runtime wired (audit #5) Pre-v3.53 ``MCPAdapter.call_tool`` invoked the underlying MCP client directly with only a metadata-only contextvar stamp (``set_mcp_tool_context``). Any agentic loop calling ``adapter.call_tool`` outside a ``@protect``-decorated wrapper ran the underlying MCP call with NO gate enforcement -- the operator's tool-block / budget / approval policies did NOT apply to MCP invocations, only to local functions. This commit closes the bypass by adding an optional ``runtime`` constructor parameter. When provided, ``call_tool`` invokes ``runtime.execute(...)`` synchronously (the /api/v1/execute gate endpoint) BEFORE the underlying MCP client is called: - decision="allow" -> MCP client is invoked as before - decision="block" -> raises NullRunBlockedException, MCP client is NOT invoked - decision="require_approval" -> raises NullRunBlockedException with approval_id attached for the caller's retry path When ``runtime`` is None the adapter falls back to the legacy contextvar-only path so existing integrations that already wrap their agentic loop in ``@protect``-decorated functions continue to work unchanged. New integrations should pass ``runtime=`` so the tool-block / budget / approval policies actually apply. Changes: - src/nullrun/toolbox/mcp.py: MCPAdapter.__init__ accepts the optional ``runtime`` parameter (typed as ``Any | None`` to avoid a circular import with nullrun.runtime at module load); stores it as ``self._runtime``. ``call_tool`` invokes ``self._runtime.execute(tool_name=..., input_data=..., mode="strict")`` when wired, BEFORE the MCP client. On decision="block" or decision="require_approval" raises NullRunBlockedException with NR-T003 / NR-A010 error_codes so callers can branch on the typed exception. Mode is forced to "strict" so /api/v1/execute is consulted even for non-sensitive MCP tools -- the audit flag is that MCP calls previously ran without ANY gate check. - tests/test_mcp_adapter.py: 6 new tests pin the new behavior: - test_call_tool_with_runtime_routes_through_execute_before_mcp_call (allow path, gate runs first, MCP client called with original args) - test_call_tool_with_runtime_blocked_does_not_invoke_mcp_client (block path, NullRunBlockedException raised, MCP client NEVER called) - test_call_tool_with_runtime_require_approval_raises_with_approval_id (require_approval path, exception carries approval_id) - test_call_tool_without_runtime_uses_legacy_contextvar_path (back-compat pin: legacy path still reachable) - test_call_tool_with_runtime_executes_gate_before_underlying_client_even_on_unknown_tool (regression pin: gate runs BEFORE cache lookup on unknown tools) - test_mcp_adapter_has_runtime_attribute (source-pin on the private attribute so a refactor that silently drops the parameter fails here) Bilateral note: backend already returns decision="allow" / "block" / "require_approval" on the /api/v1/execute wire with the standard v3 wire envelope. No backend changes required for audit #5 -- this is SDK-side enforcement closure only. Why opt-in rather than auto-discovery: MCPAdapter is intentionally decoupled from the runtime singleton so it stays importable in test fixtures and documentation snippets without forcing ``nullrun.init()``. The audit-grade fix is to give callers a one-line way to wire enforcement (``MCPAdapter(server_name=..., mcp_client=conn, runtime=nullrun.get_runtime())``) without breaking the toolbox-only pattern. 1536 passed, 7 skipped (no regressions). * fix(sdk): refuse NULLRUN_SKIP_BUDGET_CHECK=1 in production (v3.53 audit #6) Pre-v3.53 the SDK silently honored `NULLRUN_SKIP_BUDGET_CHECK=1` regardless of environment. CLAUDE.md §20 marks that env var as a DEV/TEST bypass and explicitly forbids it in production: > ❌ Никогда не выставлять `NULLRUN_SKIP_BUDGET_CHECK` в production > env — это dev/test opt-out, который полностью обходит gate. The pre-v3.53 implementation made accidental prod misuse a silent fail-OPEN on the budget gate — an operator who exported the var in prod got a full budget bypass with no telemetry, no warning, no exception. Fix shape (v3.53 audit #6): 1. New `_is_production_environment(api_url)` helper in runtime.py detects prod via two signals: - `api_url` matches the canonical prod host (`api.nullrun.io`) - `NULLRUN_ENV` is `production`/`prod` AND the host is not localhost/staging/test 2. `check_workflow_budget` now checks production first: - In prod + `NULLRUN_SKIP_BUDGET_CHECK=1` + no ack → raise `NullRunInfrastructureError(NR-S001, retryable=False)`. Emits `skip_budget_blocked_in_prod` metric. - In prod + `NULLRUN_SKIP_BUDGET_CHECK=1` + `NULLRUN_ALLOW_SKIP_BUDGET_CHECK=1` → warn log + `skip_budget_allowed_in_prod` metric + skip. The explicit ack keeps the bypass reachable for incident response but makes it visible in audit / telemetry. - In dev/test → silent skip (legacy behavior preserved). 3. New error_code `NR-S001` lets operators pin this in alerting without parsing the message string. Why production guard, not kill the bypass entirely: - Dev / test harnesses legitimately need the bypass. - The previous CLAUDE.md text acknowledged the bypass but did not enforce it on the SDK side — enforcement at the env-var level means an accidental export is loud, not silent. - The explicit ack path (`NULLRUN_ALLOW_SKIP_BUDGET_CHECK=1`) mirrors the existing `NULLRUN_SENSITIVE_FAIL_OPEN=1` pattern: same shape, same warning log, same metric increment. Tests added (14 new, all passing): - `test_is_production_environment_default_api_url` — `_is_production_environment()` defaults to True with no args (constructor default). - `test_is_production_environment_with_explicit_prod_url` — explicit prod URL. - `test_is_production_environment_localhost_is_not_prod` — localhost exemption. - `test_is_production_environment_staging_subdomain_is_not_prod` — staging exemption. - `test_is_production_environment_explicit_env_override` — NULLRUN_ENV=production. - `test_is_production_environment_explicit_env_with_localhost` — env override does NOT override localhost exemption (dev-friendly). - `test_is_production_environment_prod_alias` — "prod" alias. - `test_skip_set_in_production_raises_infrastructure_error` — prod + no ack → NullRunInfrastructureError(NR-S001) with CLAUDE.md §20 reference. - `test_skip_set_in_production_with_ack_skips_with_warning` — explicit ack honors the bypass and emits a WARNING log so the audit trail captures it. - `test_skip_set_in_dev_skips_silently` — dev/test URL → silent skip. - `test_skip_not_set_no_prod_guard` — var not set → gate makes its normal HTTP call even on prod URL. - `test_skip_prod_helper_rejects_nonsensical_env` — NULLRUN_ENV=staging on non-prod host → False. - `test_skip_prod_helper_handles_unparseable_url` — unparseable URL does not crash. - `test_skip_prod_helper_lowercases_hostname` — `API.NULLRUN.IO` matches. Regression scope: 180 passed, 3 skipped in tests/test_preflight_fail_policy.py + test_no_local_policy.py + test_transport.py + test_transport_branches.py. No regressions. Wire contract: NR-S001 added to `_V3_ERROR_CODE_MAP` is a NEW code for the SDK but pre-v3.53 SDKs do not raise it (silent skip), so the convention is purely additive. Audit cross-references: - v3.53 audit #6 (skip-budget-check production enforcement) - CLAUDE.md §20 (security opt-outs in production forbidden) - CLAUDE.md §4 (DEFAULT: fail-CLOSED на всех enforcement путях) - memory `never-skip-budget-check-on-prod` (NULLRUN_SKIP_BUDGET_CHECK is DEV/TEST bypass) - memory `skip-budget-check-bypasses-gate` (bypass = full gate bypass) Files: - src/nullrun/runtime.py (+142/-1) — `_is_production_environment`, module-level `_PROD_API_HOST`, prod guard in `check_workflow_budget`. - tests/test_preflight_fail_policy.py (+251/0) — new `TestSkipBudgetCheckProductionGuard` class with 14 tests. * chore(release): 0.15.1 — v3.53 audit closure (H6/L5/L6/M8 + #4/#5/#6) Patch release bundling the v3.53 NULLRUN audit fixes that landed between 0.15.0 and now. Six fixes land on the wire path: - audit #4: Transport.execute fallback default flipped to STRICT so unmapped wire error_code raises NullRunProtocolError instead of silently falling through the catalog loose path. - audit #5: MCPAdapter.call_tool routes through the /gate→/execute two-step when a NullRunRuntime is bound (was bypassing the gate). - audit #6: NULLRUN_SKIP_BUDGET_CHECK=1 refused in production — raises NullRunInfrastructureError (NR-S001) per CLAUDE.md §20. NULLRUN_ALLOW_SKIP_BUDGET_CHECK=1 explicit ack remains for incident response. - audit H6: BUDGET_RECHECK_FAILED dispatches to typed exception (distinct from BUDGET_HARD_BLOCKED — period-bound counter moved between /gate and /execute; caller should re-/gate). - audit A-1+A-2: six approval grant-consume outcomes (APPROVAL_NOT_YET_APPROVED / DENIED / EXPIRED / DIGEST_MISMATCH / TOOL_DIGEST_MISMATCH / REPLAY_REJECTED) get typed NR-A010..NR-A015 dispatch — was collapsing to NullRunBlockedException which silently crashed on the loose path because subclasses need workflow_id positional. - audit M8: _validate_capabilities_payload rejects malformed capability envelopes at SDK entry rather than passing them downstream. Static-typing closure: - _V3_ERROR_CODE_MAP annotation tightened from type[BaseException] to type[Exception] (mypy return-value fix — every map value is Exception subclass). - ruff F811 sweep across test files (test_actions.py, test_v3_wire_contract.py, test_audit_wire.py, test_no_local_policy.py, test_audit.py, test_runtime.py, test_transport.py) — auto-removed redefinition of unused top-level imports shadowed by in-function imports. - runtime.py non_prod_hosts tuple: 0.0.0.0 is a host-marker string for the substring match, not a bind address — silenced S104 with noqa rationale. Tests: 1550 passed, 7 skipped in 154.47s. ruff clean. mypy clean (37 source files). Compatibility: No SDK_MIN_VERSION bump. No public API change, no wire-format change. Drop-in replacement for 0.15.0.
maltsev-dev
added a commit
that referenced
this pull request
Aug 14, 2026
…re (#90) * test(sdk): remove flaky test_env_fallback_when_server_value_is_zero The push-CI coverage job on 0.15.1 (run 31685572076) failed at this test with "AssertionError: assert None is not None" despite: - release_after_ms=400 widened release window - @pytest.mark.rerunfailures(reruns=4) on inner helper Root cause: @pytest.mark.rerunfailures only reruns TOP-LEVEL pytest test functions. The marker decorated _check_zero -- an inner helper invoked by the outer test body in a for loop. Pytest never collected the marker, so reruns never fired. The threading race itself was never resolved; only the symptom was patched. Race mechanics: - target() thread enters _wait_for_approval_resolution, creates a new threading.Event(), calls event.wait(timeout). - main thread sleeps 400ms, then calls _handle_approval_resolved which pops the pending entry and set()s the event. - If main releases BEFORE target reaches event.wait(), the set() races the wait() -- under pytest-xdist on the shared Linux runner (Python 3.12), target sometimes misses the window, event.wait(timeout_seconds=120) blocks, and the 5s t.join timeout fires before the 120s wait releases. Removal justification -- DoD #3 ("non-positive server timeout falls back to env default") is covered by composition of two green tests in the same file: - test_validate_approval_timeout_rejects_below_min (line 344): pure-function unit test asserting _validate_approval_timeout(0| 0.0|-1|-100|0.99) is None. Deterministic, never flaky. - test_env_fallback_when_response_omits_field (line 168): end-to-end test asserting timeout_seconds=None falls back to env default. Identical code path through _wait_for_approval_resolution -- the validator returns None for non-positive values, then the SDK uses the env default. Test file: tests/test_approval_timeout_field.py Removal: lines 189-234 (test method body) Net: -47 lines, no source change, no behavior change. Verification: - Local pytest: 1549 passed, 7 skipped (was 1550+7; -1 expected). - ruff clean. mypy clean (37 source files). - 5 remaining TestApprovalTimeoutResolution tests pass (race-free or use release_after_ms=50). - 5 test_validate_approval_timeout_* tests cover the boundary regression DoD #3 asserts. * fix(sdk): observability closure on check_workflow_budget fail-OPEN paths The fail-OPEN posture on SDK transport failure is the documented ADR-008 contract (top-of-runtime.py table) and is unchanged by this commit. Pre-0.15.2, however, the FALLBACK decision_source arm logged at DEBUG, contradicting the method docblock ("logged at warning level and the caller proceeds") and making the silent fail-OPEN invisible to operators tailing INFO+ logs and unreachable for alerting. Promote logger.debug to logger.warning on the synthetic FALLBACK path (runtime.py:2017) and add metrics.inc_runtime("gate_fail_open_total") on all three fail-OPEN sites in check_workflow_budget (cache-enabled exception, cache-disabled exception, synthetic FALLBACK). Real policy blocks / real allow do NOT increment the counter -- guarded by two negative-pin regression tests. New RuntimeMetrics.gate_fail_open_total field (observability/__init__.py) exposed via metrics.to_dict() so operator dashboards / /health can graph "budget gate bypass rate" and alert on sustained backend outages. 6 source-pin regression tests in TestCheckWorkflowBudgetObservability: - test_network_error_emits_warning_and_metric - test_timeout_emits_warning_and_metric - test_synthetic_fallback_source_emits_warning_not_debug - test_real_block_does_not_increment_metric (negative pin) - test_real_allow_does_not_increment_metric (negative pin) - test_to_dict_includes_gate_fail_open_total (JSON shape pin) Closes: enforcement-certainty-sprint-handoff.md (Bug #4, HIGHEST severity) Test count: 1556 passed (+6), 7 skipped. ruff clean. * @ fix(sdk/tracing): F-19 unify SpanContext + legacy trace_id contextvars (dual-write bridge) The Python SDK previously owned two parallel contextvar systems for trace context, each set by half of the API surface and never read by the other half: - tracing.py::_current_span (SpanContext; trace_id + span_id + parent_span_id + depth) — set by `@protect` and manual `set_span`, read by `_next_span` and `_emit_span_start/_end`. - context.py::_trace_id_var / _span_id_var — set by `with workflow(...)` and `with span(...)`, read by `runtime._enrich_event` (cost-event trace_id / span_id / parent_trace_id). Result (`UI-UX-AUDIT-REPORT.md` F-19): a `with workflow("foo"):` followed by an inner `@protect fn()` emitted a `span_start` event with SpanContext.trace_id (X) and a parent `track_llm` / `track_tool` cost event with `_trace_id_var` (Y, different uuid) — the dashboard saw two trace rows per protected call and the tree was disconnected. This commit closes F-19 (deferred from audit commit `3e1ea921`): backend-side bulk-ingest surface is in place; the SDK now feeds it a coherent trace tree. Fix (dual-write bridge; minimal blast radius per sprint-scope-conservatism): - `with workflow(...)` — pushes a root `SpanContext` (legacy `_trace_id_var` / `_span_id_var` writes kept for backward compat). New token-based `reset_span(...)` paired with the legacy resets in `finally`. - `with span(...)` — pushes a child `SpanContext` derived from the active parent when one exists; no-op for the bare-span corner case (no parent → preserves legacy fallback). - `@protect` `_protect_body` — after `set_span(span)`, mirrors `span.trace_id` / `span.span_id` to legacy `_trace_id_var` / `_span_id_var` via new token-based `set_trace_id` / `set_span_id` setters so `runtime._enrich_event` reads the SAME trace id for both span_start and cost events. `finally` resets all four tokens in lockstep. Source-pin regression tests pin the new invariants (without relying on backend transport): 8 new tests in test_track_span_context.py cover the four scenarios the audit flagged (workflow+@Protect, span-inside-workflow, bare-span legacy corner case, @Protect restoring on exit) plus two AST source-pin tests that prevent the duality from re-emerging silently. Verification: - 19/19 tests in test_track_span_context.py pass (11 pre-existing + 8 new F-19 source-pin regressions). - 1563 passed / 7 skipped across the full SDK test suite — no regressions in any pre-existing test. - `ruff check` and `ruff format --check` clean on the three changed files. Wire / contract preservation: - No public API change: `nullrun.workflow`, `nullrun.span`, `get_trace_id`, `get_span_id`, `get_current_span`, `set_span` / `reset_span`, etc. all keep their existing signatures and semantics. - The legacy contextvars remain readable (used by `runtime._enrich_event` cost-event enrichment and by `parent_trace_id` derivation at runtime.py:2967). - `@protect` / `with workflow` / `with span` consumers see no behavior change for the legacy readers; the only new observable is that the SpanContext (read via `get_current_span()`) and the legacy vars now agree on `trace_id` / `span_id` at every nesting level. @ * fix(instrumentation/langgraph): F-28 threading.RLock protects _active_runs UI-UX-AUDIT 2026-08-14 finding F-28: NullRunCallback._active_runs is read/written without synchronisation on multi-threaded LangChain runners (and on free-threaded CPython PEP 703 builds). Two callbacks on different threads can interleave on_chain_start / on_chain_end in ways that orphan the span_end lookup (parent_span_id on the wire doesn't match anything in the dict). Fix: wrap every read/write of _active_runs in with self._lock: (threading.RLock) RLock (not Lock) is required because _begin_run -> _register_active_run nests two acquisitions on the same thread — reentrant acquisition is the entire point. Five access sites wrapped: 1. _register_active_run (insert + cap-check eviction) 2. on_llm_start parent_ctx lookup 3. on_llm_end llm_ctx lookup 4. _begin_run parent_ctx lookup 5. _end_run pop Trade-off: the lock briefly spans runtime.track_event. Per callback that's one outbound HTTP round-trip holding the lock; acceptable because the Lock protects ONE NullRunCallback's dict (not all of them) and concurrent chains on the SAME callback are rare. Documented inline at __init__ so a future maintainer doesn't 'optimise' it away. Regression: tests/test_langgraph_callback_race.py (5 tests): - test_active_runs_lock_is_rlock : reentrant acquire from this thread - test_active_runs_protected_under_concurrent_register : 200 iter, 2 threads, cap=64 - test_active_runs_protected_under_register_end_race : register + pop race - test_active_runs_lock_does_not_deadlock_on_nested_register : nested acquire - test_register_then_end_round_trip : canonical happy-path sanity Verification: pytest tests/test_langgraph_callback_race.py tests/test_lru_active_runs.py tests/test_langgraph_callback.py -q: 54 passed ruff check src/nullrun/instrumentation/langgraph.py tests/test_langgraph_callback_race.py: All checks passed pytest tests/ -q (excluding integration): 1568 passed, 7 skipped * fix(instrumentation/auto): F-29 async _emit falls back to request-body model UI-UX-AUDIT 2026-08-14 finding F-29: NullRunAsyncTransport._emit stopped at usage.get('model') only — when the upstream Anthropic or OpenAI streaming response omitted a top-level model field, the emitted llm_call event had model=None, which the wire-format builder dropped, which the backend then unwrap_or('default')'d to DEFAULT_RATE. Net effect: silent zero-billing for async streaming clients. Fix: mirror the sync path's fallback chain at auto.py:882-885: model_for_event = ( usage.get('model') or _extract_model_from_request_body(request) ) _extract_model_from_request_body is a module-level pure-sync helper that reads request.content + json.loads — safe to call from the async event loop (no I/O, no blocking). The response body is tried first; the request body is the fallback when the response omits the field. The pre-fix comment at lines 967-971 explicitly noted 'async path doesn't have the request-body model fallback yet' — that comment is now stale and replaced with F-29 context. Regression: tests/test_model_fallback_async.py (3 tests): - test_async_transport_falls_back_to_request_body_model : main F-29 case - test_async_transport_prefers_response_body_model : response wins when both - test_async_transport_emits_none_when_neither_source_has_model : corner case Verification: pytest tests/test_model_fallback.py tests/test_model_fallback_async.py tests/test_streaming_oom_cap.py: 17 passed ruff check src/nullrun/instrumentation/auto.py tests/test_model_fallback_async.py: All checks passed pytest tests/ -q (excluding integration): 1571 passed, 7 skipped * chore(release): 0.15.2 — observability + UI-UX-AUDIT 2026-08-14 closure Patch release bundling the 5 commits accumulated since 0.15.1: - 1c96654 — check_workflow_budget fail-OPEN observability closure (sprint handoff Bug #4): synthetic FALLBACK path now logs at WARNING (not DEBUG), and a new gate_fail_open_total counter fires on all three fail-OPEN sites. - 9b87d20 — F-19 SpanContext ↔ legacy trace_id/span_id contextvars now form a single coherent trace tree. Pre-0.15.2 inner @Protect fn() inside a with workflow("foo") block emitted two disconnected trace rows on the dashboard. - 127b003 — F-28 NullRunCallback._active_runs now protected by threading.RLock; five access sites wrapped, reentrant for _begin_run → _register_active_run nesting. - 0f86c8c — F-29 NullRunAsyncTransport._emit now falls back to the request body model field when the upstream Anthropic / OpenAI streaming response omits it. Closes silent-zero-billing bug for async streaming clients. - 35728b9 — removed flaky test_env_fallback_when_server_value_is_zero; the contract is covered by composition of test_validate_approval_timeout_rejects_below_min + test_env_fallback_when_response_omits_field (both deterministic). Verification: - pytest: 1571 passed, 7 skipped in 103.85s - ruff: clean - mypy: clean (37 source files) Compatibility: No SDK_MIN_VERSION bump. No public API change. No wire-format change. Drop-in replacement for 0.15.1.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The SDK's import chain (nullrun.init -> nullrun.decorators -> nullrun.instrumentation.langgraph -> 'from langchain_core.callbacks import BaseCallbackHandler') runs at pytest collection time, not at a specific test. With CI installing [dev] only, every test in the suite errored on collection with:
This is the same class of bug that 'nullrun[langgraph]' exists to prevent for end users, except the dev install never benefited from the extras indirection.
Fix: add 'langchain-core>=0.3,<1.0' to the [dev] extras. The heavier 'langgraph' / 'langchain' extras pull in stacks the unit tests don't use; the bare core is the smallest dep that makes the import chain resolve and unblocks test collection on every supported Python (3.10 / 3.11 / 3.12) on every PR.
Validation: locally on Python 3.14.2 (which is outside the 3.10/3.11/3.12 matrix that CI tests), 'pip install -e .[dev]' followed by 'pytest tests/' runs 443/443 + 9/9 new byte-mismatch unit tests, no collection error. CI will re-confirm on the 3.10 / 3.11 / 3.12 matrix.
What
Why
How
Test plan
cd backend && cargo test,cd frontend && npm test)cd frontend && npm run lint)cd frontend && npm run type-check)Risk
Checklist
CONTRIBUTING.md(if present)