Add read-only local backend metadata probe (CR-114)

docs/change/requests/CR-114-local-backend-metadata-probe.md

@@ -0,0 +1,109 @@
+# CR-114: Read-Only Local Backend Metadata Probe
+
+## Status
+
+Implemented in branch; pending review/merge
+
+Implements the read-only local backend metadata probe slice bounded by the
+CR-113 design brief (`docs/operations/local-backend-telemetry.md`). This is the
+first non-deterministic contact point in the runtime-strategy/telemetry lane: it
+observes local backend availability and model/runtime identity via metadata
+endpoints only. It adds no model generation, no routing changes, no
+`ResilienceRouteInput` wiring, no circuit breakers, no background polling, no
+ledger writes, and no default artifact writes. It is also the observation
+foundation the daily-driver M1 milestone will later consume.
+
+## Scope
+
+- Add `triage_core/local_backend_probe.py` providing:
+  - a metadata-only `LocalBackendProbeRecord` (closed `source_type`,
+    `error_category`, and `evidence_tier` vocabularies) whose `to_dict()`
+    enforces the persistent privacy invariant on every emitted record;
+  - `redact_base_url()` (validation rule: reject userinfo/query/fragment, strip
+    path, store `scheme://host[:port]` only);
+  - `probe_local_backend()` which hits a metadata endpoint only
+    (`ollama -> /api/tags`, `lm_studio` / `llama_cpp -> /v1/models`), fails
+    closed with a closed `error_category`, and accepts an injectable transport
+    for offline testing;
+  - `render_probe_record()` for a privacy-safe human-readable summary.
+- Add a `tc probe` command:
+  `tc probe --source-type {ollama|lm_studio|llama_cpp} --base-url <url>`
+  with `--timeout`, `--include-model-names`, `--disabled`, and `--output`.
+- Add offline tests in `tests/test_local_backend_probe.py`.
+- Update `docs/operations/local-backend-telemetry.md` Status to record that
+  CR-114 implements the probe slice only.
+
+### Exit codes
+
+- `0` — the probe produced a valid metadata record, including `reachable=false`
+  records and `probe_disabled` records.
+- `1` — argument / input / validation error, including a secret-bearing
+  `base_url` (userinfo or query present).
+
+There is no exit `2` for this slice; there is no config/policy gate that makes
+the command unable to run. `--disabled` is a valid record outcome (exit `0`),
+not a crash.
+
+## Non-Goals
+
+- No model generation, completions, or embeddings; metadata endpoints only.
+- No cloud/frontier probing; local endpoints only, nothing leaves the machine.
+- No routing changes and no `ResilienceRouteInput` wiring (future M1 slice).
+- No circuit breakers or degraded-mode states (future M1 slice).
+- No background polling and no automatic discovery; probing is opt-in and
+  explicit against an operator-supplied URL.
+- No ledger writes and no default artifact write location; only `--output`
+  writes an operator-named file.
+- No new dependencies beyond the existing `requests`.
+- No claim that the full telemetry design is implemented; only the probe slice
+  is built.
+
+## Description
+
+The runtime-strategy lane is otherwise deterministic. The CR-113 brief bounded a
+future read-only probe as the point where that determinism boundary is crossed,
+fixing the record shape, closed failure vocabulary, redaction rules,
+evidence-tier provenance, opt-in posture, and reviewer path before any probe
+code existed. CR-114 implements exactly that slice and nothing beyond it.
+
+The probe observes; it does not act. It never feeds routing, never mutates
+state, and never invokes a model. `base_url` redaction is enforced as a
+validation rule (secret-bearing URLs are rejected, not stored), path-like model
+identifiers are dropped, `observed_models` is off by default, and every emitted
+record passes the persistent privacy invariant. Fixture-tier records carry no
+timestamp so deterministic exports stay byte-identical; only probe/recorded
+tiers may carry `observed_at`.
+
+## Acceptance Criteria
+
+- [x] `probe_local_backend()` returns a metadata-only record and never invokes a
+  model endpoint.
+- [x] Closed vocabularies are enforced for `source_type` (via the CLI choices
+  and an `unsupported_backend` record), `error_category`, and `evidence_tier`.
+- [x] `base_url` is stored redacted as `scheme://host[:port]`; userinfo/query
+  URLs are rejected with an input error (exit `1`); paths are stripped.
+- [x] `observed_models` is off by default; path-like identifiers are dropped
+  when names are requested.
+- [x] Every emitted record passes `assert_persistent_privacy_safe`.
+- [x] `tc probe` exits `0` for valid records (including `reachable=false` and
+  `probe_disabled`) and `1` for argument/validation errors.
+- [x] `--output` is the only write path; there is no default artifact and no
+  ledger write.
+- [x] Tests run offline (injected transport plus a closed local port), covering
+  each failure category, redaction/rejection, determinism, and the privacy
+  invariant.
+
+## Validation
+
+- `python -m pytest -q tests/test_local_backend_probe.py`
+- `tc audit --privacy-invariants` (CR-021 invariant remains clean)
+- `git diff --check`
+- Manual (optional): `tc probe --source-type ollama --base-url http://localhost:11434`
+  against a live local backend.
+
+## Dependencies / Sequencing
+
+- Implements the CR-113 telemetry design brief's probe slice.
+- Precedes the M1 slices that wire probe output into `ResilienceRouteInput`,
+  bind `local_fast`/`local_heavy` to distinct models, and add circuit breakers /
+  degraded modes. None of those are started here.

docs/operations/local-backend-telemetry.md

@@ -16,9 +16,16 @@ and privacy rules agreed before any probe code is written.
 
 ## Status
 
-Design brief only. No telemetry code, CLI surface, schema module, or fixture
-exists for this lane yet. Nothing in this document describes current
-behavior; every statement below is future design intent.
+Partially implemented. **CR-114 implements the read-only metadata probe slice**
+only: the `triage_core/local_backend_probe.py` module and the `tc probe` command
+produce metadata-only records against the endpoints below, with the closed
+failure vocabulary, tiered evidence, and `base_url` redaction described here.
+
+Routing wiring, route-input population (`ResilienceRouteInput`), circuit
+breakers, degraded modes, and any daily-driver enforcement **remain future
+work** and are out of scope for CR-114. This document still describes the full
+telemetry design; only the read-only probe slice is built. Nothing here should
+be read as implying the whole telemetry design is implemented.
 
 ## Supported Future Sources
 

tests/test_local_backend_probe.py

@@ -0,0 +1,248 @@
+"""Offline tests for the read-only local backend metadata probe (CR-114).
+
+All tests run without a live backend. Network outcomes are exercised through an
+injected transport or a closed local port; no test contacts a real model
+endpoint, and no test writes to the real ledger.
+"""
+import json
+import socket
+
+import pytest
+import requests
+
+from triage_core.local_backend_probe import (
+    ERROR_CATEGORIES,
+    LocalBackendProbeRecord,
+    ProbeInputError,
+    probe_local_backend,
+    redact_base_url,
+    render_probe_record,
+)
+from triage_core.privacy_invariants import find_forbidden_persistent_fields
+
+
+# ---- transports (injected) -------------------------------------------------
+
+def _ollama_ok(url, timeout):
+    return 200, {"models": [{"name": "qwen2.5-coder:7b"}, {"name": "llama3.1:8b"}]}
+
+
+def _openai_ok(url, timeout):
+    return 200, {"data": [{"id": "local-model"}, {"id": "another-model"}]}
+
+
+def _path_like_models(url, timeout):
+    return 200, {
+        "models": [
+            {"name": "qwen2.5-coder:7b"},
+            {"name": "/home/corey/models/private.gguf"},
+        ]
+    }
+
+
+def _malformed(url, timeout):
+    return 200, {"unexpected": "shape"}
+
+
+def _timeout(url, timeout):
+    raise requests.exceptions.Timeout("timed out")
+
+
+def _refused(url, timeout):
+    raise requests.exceptions.ConnectionError("connection refused")
+
+
+def _blocked(url, timeout):
+    raise PermissionError("blocked by policy")
+
+
+# ---- reachable / model counting -------------------------------------------
+
+def test_reachable_ollama_counts_models_no_names_by_default():
+    rec = probe_local_backend(
+        source_type="ollama",
+        base_url="http://localhost:11434",
+        transport=_ollama_ok,
+    )
+    assert rec.reachable is True
+    assert rec.model_count == 2
+    assert rec.observed_models is None  # off by default
+    assert rec.error_category is None
+    assert rec.evidence_tier == "local_metadata_probe"
+
+
+def test_openai_shape_backends_use_v1_models():
+    rec = probe_local_backend(
+        source_type="lm_studio",
+        base_url="http://localhost:1234",
+        transport=_openai_ok,
+    )
+    assert rec.reachable is True
+    assert rec.model_count == 2
+
+
+def test_include_model_names_drops_path_like_identifiers():
+    rec = probe_local_backend(
+        source_type="ollama",
+        base_url="http://localhost:11434",
+        include_model_names=True,
+        transport=_path_like_models,
+    )
+    assert rec.model_count == 2
+    # The path-like identifier is dropped from the recorded names.
+    assert rec.observed_models == ["qwen2.5-coder:7b"]
+
+
+# ---- fail-closed categories (all valid records, exit 0 territory) ----------
+
+def test_unsupported_backend_is_a_record_not_an_error():
+    rec = probe_local_backend(
+        source_type="vllm",  # outside the closed vocabulary
+        base_url="http://localhost:8000",
+        transport=_openai_ok,
+    )
+    assert rec.reachable is False
+    assert rec.error_category == "unsupported_backend"
+
+
+def test_disabled_probe_emits_probe_disabled_record():
+    rec = probe_local_backend(
+        source_type="ollama",
+        base_url="http://localhost:11434",
+        enabled=False,
+        transport=_ollama_ok,  # must not be consulted
+    )
+    assert rec.reachable is False
+    assert rec.error_category == "probe_disabled"
+
+
+def test_timeout_category():
+    rec = probe_local_backend(
+        source_type="ollama", base_url="http://localhost:11434", transport=_timeout
+    )
+    assert rec.reachable is False
+    assert rec.error_category == "timeout"
+
+
+def test_endpoint_unreachable_category():
+    rec = probe_local_backend(
+        source_type="ollama", base_url="http://localhost:11434", transport=_refused
+    )
+    assert rec.reachable is False
+    assert rec.error_category == "endpoint_unreachable"
+
+
+def test_permission_or_policy_blocked_category():
+    rec = probe_local_backend(
+        source_type="ollama", base_url="http://localhost:11434", transport=_blocked
+    )
+    assert rec.reachable is False
+    assert rec.error_category == "permission_or_policy_blocked"
+
+
+def test_malformed_response_category():
+    rec = probe_local_backend(
+        source_type="ollama", base_url="http://localhost:11434", transport=_malformed
+    )
+    assert rec.reachable is False
+    assert rec.error_category == "malformed_response"
+
+
+def test_unreachable_against_closed_local_port_no_injection():
+    # A genuinely closed local port: fail closed with no network fixture.
+    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+    sock.bind(("127.0.0.1", 0))
+    closed_port = sock.getsockname()[1]
+    sock.close()
+    rec = probe_local_backend(
+        source_type="ollama",
+        base_url=f"http://127.0.0.1:{closed_port}",
+        timeout=0.25,
+    )
+    assert rec.reachable is False
+    assert rec.error_category in {"endpoint_unreachable", "timeout"}
+
+
+# ---- base_url redaction / secret rejection --------------------------------
+
+def test_base_url_redacts_path_to_scheme_host_port():
+    assert redact_base_url("http://localhost:11434/api/tags") == "http://localhost:11434"
+    assert redact_base_url("http://localhost:11434/some/path") == "http://localhost:11434"
+    assert redact_base_url("http://localhost") == "http://localhost"
+
+
+def test_base_url_with_userinfo_is_rejected():
+    with pytest.raises(ProbeInputError):
+        redact_base_url("http://user:secret@localhost:11434")
+
+
+def test_base_url_with_query_is_rejected():
+    with pytest.raises(ProbeInputError):
+        redact_base_url("http://localhost:11434/v1/models?token=abc")
+
+
+def test_probe_raises_input_error_for_secret_bearing_url():
+    with pytest.raises(ProbeInputError):
+        probe_local_backend(
+            source_type="ollama",
+            base_url="http://user:pass@localhost:11434",
+            transport=_ollama_ok,
+        )
+
+
+# ---- privacy invariant / determinism --------------------------------------
+
+def test_every_record_passes_persistent_privacy_invariant():
+    rec = probe_local_backend(
+        source_type="ollama",
+        base_url="http://localhost:11434",
+        include_model_names=True,
+        transport=_ollama_ok,
+    )
+    assert find_forbidden_persistent_fields(rec.to_dict()) == []
+
+
+def test_synthetic_fixture_tier_forbids_timestamp():
+    # Fixture-tier records stay byte-identical: no observed_at allowed.
+    ok = LocalBackendProbeRecord(
+        source_type="ollama",
+        base_url="http://localhost:11434",
+        reachable=True,
+        evidence_tier="synthetic_fixture",
+        model_count=1,
+    )
+    assert ok.observed_at is None
+    with pytest.raises(ValueError):
+        LocalBackendProbeRecord(
+            source_type="ollama",
+            base_url="http://localhost:11434",
+            reachable=True,
+            evidence_tier="synthetic_fixture",
+            model_count=1,
+            observed_at="2026-07-09T00:00:00+00:00",
+        )
+
+
+def test_error_category_vocabulary_is_closed():
+    with pytest.raises(ValueError):
+        LocalBackendProbeRecord(
+            source_type="ollama",
+            base_url="http://localhost:11434",
+            reachable=False,
+            evidence_tier="local_metadata_probe",
+            error_category="something_new",
+        )
+    assert "endpoint_unreachable" in ERROR_CATEGORIES
+
+
+def test_render_is_readable_and_privacy_safe():
+    rec = probe_local_backend(
+        source_type="ollama",
+        base_url="http://localhost:11434",
+        transport=_ollama_ok,
+    )
+    text = render_probe_record(rec)
+    assert "source_type:" in text
+    assert "reachable:" in text
+    # Rendering must not leak forbidden content (it is built from to_dict()).
+    assert "secret" not in text.lower()