Delegation preflight: check the endpoint before you delegate, and fix two probes that were understating reality

docs/DIVERGENCE_2026-07-31.md

@@ -0,0 +1,121 @@
+# Divergent production state — incident record, 2026-07-31
+
+Baseline: `e5b9130` / v2.0.2, deployed 2026-07-29T14:11:33Z.
+
+## 1. What was observed
+
+Two consecutive daily ops passes saw production serve two mutually
+inconsistent views of the same counters:
+
+| surface | early reads | later reads (stable) |
+|---|---|---|
+| `/instrumentation` `total_events` | 13344 | 16796 |
+| `/instrumentation` `genuine_external.total_events` | 222 | 242 |
+| `/instrumentation` `genuine_external.passports_issued` | 1 | 3 |
+| `/funnel/passports` `offer_served` | 944 (108 actors) | 1788 (226 actors) |
+| `POST /ledger/checkpoint/publish` | index **14** / ledger_length **834** | index 16 / 836 |
+
+The `13344 / 222 / 1` triple was byte-identical on 07-30 and 07-31 — a
+**frozen** view, not a lagging one. On 07-31 it survived three consecutive
+reads spaced ~10s apart before flipping.
+
+The checkpoint case is the serious one: the **write path** was willing to act
+on a view two entries behind the committed feed.
+
+## 2. What was tested, and what it ruled out
+
+| test | result | rules out |
+|---|---|---|
+| 40 concurrent `GET /release` | ONE `_PROCESS_STARTED_AT` (`2026-07-29T14:11:33.735141Z`) | a second origin *at observation time*; also proves the process had not restarted |
+| 60 concurrent mixed-endpoint requests, body-shape checked against the requested path | 0 mismatches | cross-request response mixing (e.g. the x402 buffering middleware leaking bodies) |
+| repeated reads with a unique `?cb=` per request; `cf-cache-status: DYNAMIC`, `x-render-origin-server: uvicorn` | flip still reproduced | URL-keyed CDN caching |
+| code review of the read path | `/instrumentation` and `/funnel/passports` compute from `Store.events`, a plain in-process list; `Store` is a single module-level singleton in `app/state.py`; `_load()` is called only from `__init__`; the container runs one uvicorn process with no `--workers` | an in-process reload or a second `Store` inside the process |
+
+## 3. Root cause — what is and is not proved
+
+**Read side: NOT PROVED. Deliberately not "fixed".**
+
+Every remaining candidate — a second serving origin that the concurrency probe
+never happened to hit, or an intermediary serving a body this process did not
+produce — is equally consistent with the evidence, because **no response
+carried anything identifying which process or which state produced it**. A
+speculative state fix would have been a guess dressed as a remedy.
+
+**Write side: PROVED, and reproduced locally.** Independent of whatever caused
+the read flip, `publish_checkpoint` had four defects on the canonical
+commitment path, all reproducible with two `Store` instances over one shared
+SQLite file (`tests/test_state_divergence.py`):
+
+1. **TOCTOU.** The authoritative `all_ledger()` / `all_checkpoints()` read
+   happened *before* the append, outside the write transaction. Two publishers
+   could compute the same next index.
+2. **Silent overwrite.** `put_checkpoint` used `INSERT OR REPLACE` on
+   `checkpoints(idx PRIMARY KEY)`. A colliding index **replaced** a published,
+   third-party-pinned commitment, breaking the `prev_entry_sha256` chain with
+   no error.
+3. **Index from `len()`.** `index = len(self.checkpoints)` re-issues an
+   existing index if the feed has any gap.
+4. **No read-after-write.** A publish that never landed was still reported
+   `{"status": "published"}`.
+
+Separately, `SqliteBackend._commit` could drive the **thread-local** connection
+depth negative: a nested transaction that raised called `_rollback` (depth→0,
+rolling back the outer transaction too), then the outer `__exit__` called
+`_commit` (depth→−1). From then on that thread skipped `BEGIN IMMEDIATE`, so
+its writes silently ran in autocommit and `in_transaction()` lied to
+`Store._save`. Connections are per-thread, so one poisoned request thread would
+diverge from every other thread for the life of the process.
+
+## 4. What shipped
+
+**Decidability (so the next occurrence is not a guess)**
+
+- `app/instanceid.py` — random per-process `instance` id, `boot_at`, `pid`.
+- Every response carries `X-Guild-Instance`, `X-Guild-Boot`,
+  `X-Guild-Store-Rev` (a monotonic in-memory mutation counter).
+- `GET /diagnostics/state` — in-memory counts vs **authoritative SQLite**
+  counts, with a `divergence` list naming the exact disagreement. No paths,
+  tokens, hostnames or environment are exposed.
+- `live/scripts/detect_divergence.py` — fans out concurrent reads and returns
+  one of `consistent` / `split_origin` / `stale_in_process` /
+  `memory_durable_split` / `intermediary`. **This replaces "discard the first
+  2–3 reads"**, which was a reporting workaround with no write-path coverage.
+
+**Fail-closed canonical writes**
+
+- Authoritative read moved *inside* `BEGIN IMMEDIATE`.
+- `StaleDurableStateError` when the durable feed head is behind a head this
+  process already observed, or the durable ledger is shorter than memory.
+- `insert_checkpoint_strict` — plain `INSERT`; a duplicate index raises
+  `CheckpointForkError` instead of replacing history.
+- Next index derived from `max(index) + 1`, not `len()`.
+- Read-after-write byte comparison; `CheckpointWriteVerificationError`
+  otherwise. **A publish that did not land can no longer be reported as
+  published.**
+- All three map to HTTP **409** with a stable machine-readable `code`, and the
+  body states the write did not happen.
+- `_commit` depth clamped at zero, so a nested rollback no longer poisons the
+  thread.
+
+## 5. Honest statement of the evidence
+
+**No durable loss was detected.** Every cumulative counter on the warm branch
+was greater than or equal to the previous snapshot, and the ledger head
+(index 16 / length 836) was stable across five reads and unchanged from 07-30.
+
+That is *not* the same as proving individual event continuity: there is no
+per-event durable sequence audit, so the correct phrasing is **"no durable loss
+detected"**, never "no data was lost".
+
+## 6. Remaining risk
+
+- The read-side cause is still **unknown**. It is now instrumented, not
+  resolved. If it recurs, `detect_divergence.py` names the class.
+- If the verdict comes back `split_origin`, that is a **topology emergency**:
+  SQLite lives on a single-mount Render disk and the application guard can only
+  see worker processes inside its own container, never a second instance. The
+  response is a Postgres migration review, not a code patch.
+- The fail-closed publish trades availability for integrity: under genuine
+  divergence the checkpoint feed will **stop advancing** and return 409 rather
+  than publish. That is the intended trade — a gap in the feed is recoverable,
+  a fork is not.

docs/EXPERIMENT_PREFLIGHT_2026-07-31.md

@@ -0,0 +1,139 @@
+# Experiment: delegation preflight — 2026-07-31
+
+## The blunt finding first
+
+**The current pricing model is aimed at a payer who has no money.**
+
+Agent Guild's funnel sells a free credential to autonomous agents on the theory
+that they will later pay for trust reads. Measured today:
+
+- Total x402 settled volume, **all networks, July 2026: $232,329** — down 98.9%
+  from the $20.5M November 2025 peak, on flat transaction count (~195k/day at
+  ~$0.04 each: automated loop traffic, not commerce).
+- Median x402 Bazaar listing: **2 calls and 1 unique payer per 30 days**.
+- Median earning agent: **$1.65 per 30 days**; Gini 0.97 across 316 agents.
+- Virtuals Protocol: 44,051 registered agents, **69 active in July (0.16%)**.
+- Olas: 13.97M lifetime agent-to-agent transactions → **$106,941 lifetime
+  turnover → $458 of protocol fees, ever**.
+
+5% of the entire global machine-payments market is **$11.6k/month**. Agents
+earning $1.65/month cannot buy anything. Any roadmap that monetises the supply
+side of today's machine economy is arithmetically dead, and no amount of funnel
+optimisation fixes it. The passport is still the right acquisition instrument —
+it is not a revenue instrument, and we should stop implying it will become one.
+
+## What we are NOT short of
+
+Reach. `offer_served` went 944 → 1,790 in 24 hours. All of it crawlers. The
+constraint has never been distribution volume; it is that the offer asks an
+agent to invest effort now for value later, and the agents receiving it are
+registry bots with no delegation to make.
+
+## The gap with the strongest evidence
+
+The one place where a trust decision is unavoidable, and nobody serves it:
+
+| Claim in a listing | Reality when probed | Source |
+|---|---|---|
+| 170 of 183 a2aregistry agents report `is_healthy: true` (92.9%) | **62 (33.9%) complete an A2A task** | a2aregistry API + task probe |
+| 3,913 agents serve a valid Agent Card | **42 (0.8%) sign it** | Agenstry conformance sweep |
+| 2,459 agents self-label "paid" | **141 (5.7%) return a 402** | x402/Bazaar probe |
+
+**114 agents are green and broken at the same time.** The A2A discovery
+specification states in writing that it prescribes no registry API, and
+contains no mention of signatures. x402 `exact` is a push payment: irreversible
+once executed, `payTo` bound to no legal entity, documented remedy "the seller
+sends it back". Escrow and reputation are both listed as future work.
+
+Every existing signal grades a **repository or a static card, once, at
+publication time** (Glama's grade is 70% tool-description quality; Docker
+scores the image; Anthropic reviews the submission). Nobody attests to the
+**running endpoint at call time** — which is exactly where the rug pull lives.
+
+## Ranked experiments
+
+| | Qualified reach | Friction | Time to money | Defensibility | Measurable |
+|---|---|---|---|---|---|
+| **E1 Delegation preflight** → directories, orchestrators, delegating agents | High — machine-reachable, no human in the loop | **Lowest**: one unauthenticated GET | Medium — real buyers, unproven price | High — we already have signed decisions, checkpoints, evidence classes | High |
+| E2 Runtime attestation for MCP servers → enterprise platform teams | High, **proven budget** (UpGuard $1,750/mo per 50 vendors; three security acquisitions in 12 months) | High — human sales | **Fastest to real money** | Medium | High |
+| E3 Verified behavioural evidence → AI underwriters | Very narrow (Armilla, Munich Re) | High — human, contractual | Slow | High | Medium |
+
+**Chose E1**, because it is the only one executable this session without
+spending money, contacting anyone, or launching paid — and because it is
+reversible: it is one additive read-only endpoint.
+
+**E2 is the strongest money, and we should say so plainly.** Its payer is a
+human enterprise security team, which is a direct conflict with the
+machine-only clause of the constitution. That conflict is now a decision for
+Ross, not something to be quietly resolved by preferring the weaker option.
+
+## What shipped
+
+`GET /preflight?url=…` and the `guild_preflight` MCP tool. Free, no key, no
+registration. Six checks, run live:
+
+1. `endpoint_reachable`
+2. `protocol_handshake` — a real A2A/MCP handshake, **not** merely HTTP 200
+3. `agent_card_resolves`
+4. `agent_card_signed` — presence, explicitly *not* claimed as verification
+5. `payment_claim_holds` — does an advertised paid endpoint actually 402
+6. `independent_evidence` — does the Guild hold attestation history
+
+Honesty rules enforced by tests: unknowns are reported and **excluded from the
+verdict, never averaged in**; a clean verdict still publishes its unknown
+count; absence of evidence is stated as absence, not as risk; SSRF-screened so
+it can never be used as an internal port scanner.
+
+### Two of our own defects found while building it
+
+Both were **understating** reality — the same error class as overstating
+adoption, pointed the other way:
+
+- **Chunked transfer-encoding was never decoded.** The raw body begins with a
+  hex chunk-length line, so every downstream JSON check failed. Since the card
+  check is what promotes an endpoint from `http_responsive` to
+  `recently_reachable`, *any* agent served over chunked encoding was recorded
+  as unproven. This is why `verified_reachable` read **0 for every entry in the
+  demand feed** — not because nobody was reachable, but because we could not
+  read them.
+- **A large but valid card was read as no card at all.** The probe read is
+  bounded, so verbose cards arrive truncated and fail to parse. Our own card is
+  one of them: Agent Guild's endpoint failed its own preflight until this was
+  fixed.
+
+## Instrumentation, baseline and thresholds
+
+Every call records a `preflight_run` event with target, verdict, failed and
+unknown counts, and rides the qualified cohort funnel from Phase A.
+
+**Live baseline at ship time (2026-07-31, `4c28ab8`):**
+
+- adoption-grade external passport holders: **0** (the previously reported "3"
+  were third-party fetches, one of them a schema probe)
+- verified external revenue: **$0.00**
+- qualified external actors in the passport cohort: **1**
+- preflight runs: **0** (endpoint did not exist)
+
+**Success — escalate and consider pricing:**
+- ≥ 25 preflight runs from ≥ 10 distinct genuine-external actors in 14 days, **and**
+- ≥ 3 actors returning on a later day (the retention signal, not the volume one).
+
+**Kill — stop and reallocate to E2:**
+- < 5 genuine-external runs in 14 days, or
+- > 90% of runs still unattributable crawler traffic at day 14.
+
+Both thresholds are deliberately small. The point is to learn whether *anyone
+doing a real delegation* wants this, not to accumulate impressions.
+
+## Pricing — NOT shipped, needs approval
+
+The evidence says the caller who benefits is the party whose reputation depends
+on the listing working (a directory, an orchestrator, a delegating agent), not
+the agent being checked. Comparable anchors: Riskified earns **0.237% of
+screened GMV** for a guarantee-backed signal; UpGuard charges **$1,750/mo for
+50 vendors**; Vouched KYA charges **$20–$325/mo for 1k–25k delegation checks**.
+
+No price is live and no payment configuration was touched. The proposed action
+for approval is a single change: keep preflight free to a per-caller daily cap,
+then meter it at the existing x402 price. **That is a pricing change and is not
+being made without an explicit yes.**

docs/INTERFACE.md

@@ -64,6 +64,7 @@ guild_mediated requires two-party cryptographic participation, a Guild-observed
 - `POST /credentials/verify`
 - `GET /demand/feed`
 - `POST /demand/watch`
+- `GET /diagnostics/state`
 - `GET /disputes/{case_id}`
 - `POST /disputes/{case_id}/appeal`
 - `POST /disputes/{case_id}/vote`
@@ -97,6 +98,7 @@ guild_mediated requires two-party cryptographic participation, a Guild-observed
 - `GET /offers/{offer_id}`
 - `POST /offers/{offer_id}/accept`
 - `POST /outcomes`
+- `GET /preflight`
 - `POST /providers/external/discover`
 - `GET /referrals`
 - `GET /release`
@@ -142,6 +144,7 @@ guild_mediated requires two-party cryptographic participation, a Guild-observed
 - `guild_escrow_open`
 - `guild_escrow_release`
 - `guild_passport`
+- `guild_preflight`
 - `guild_prove`
 - `guild_prove_verify`
 - `guild_record`

live/guild/app/instanceid.py

@@ -0,0 +1,55 @@
+"""Process identity — the decidability layer for state divergence.
+
+WHY THIS EXISTS (divergence incident 2026-07-30/31)
+---------------------------------------------------
+Production served two mutually inconsistent views of the same counters within
+minutes: ``/instrumentation`` returned a frozen older snapshot on several
+consecutive reads before flipping to the current one, and a ``POST
+/ledger/checkpoint/publish`` returned checkpoint index 14 / ledger_length 834
+while the published feed was already at 16 / 836.
+
+The investigation could NOT decide between the candidate causes — a second
+serving origin, an intermediary serving a stale body, or an in-process stale
+durable read — because NO RESPONSE CARRIED ANYTHING THAT IDENTIFIED WHICH
+PROCESS OR WHICH STATE PRODUCED IT. Every candidate explanation was equally
+consistent with the evidence. That is a diagnosability defect, and it is fixed
+here rather than guessed around.
+
+WHAT IS STAMPED (all non-secret, all safe to publish)
+  * ``instance``  — random per-PROCESS id, minted at import. Two different
+    values observed for one release SHA PROVE more than one serving process.
+    One value across a divergent pair DISPROVES the split-origin theory and
+    points at an intermediary or an in-process stale read.
+  * ``boot_at``   — process start (UTC). Distinguishes "restarted" from
+    "second instance" when ids differ.
+  * ``pid``       — process id inside the container. Distinguishes forked
+    workers that share a boot timestamp.
+  * ``store_rev`` — monotonic in-memory mutation counter (``Store.revision``).
+    A response whose ``store_rev`` is LOWER than one already observed from the
+    same ``instance`` is a stale in-process view; across instances it is a
+    split-brain read. Either way it becomes DETECTABLE FROM OUTSIDE.
+
+It deliberately leaks no paths, tokens, environment or hostnames — the id is
+random, not derived from anything sensitive, so a third party cannot correlate
+it back to infrastructure.
+"""
+from __future__ import annotations
+
+import os
+import secrets
+from datetime import datetime, timezone
+
+#: Random per-process identity. Minted once at import; never persisted, never
+#: derived from a hostname/path/secret.
+INSTANCE_ID: str = secrets.token_hex(6)
+
+#: Process start time (UTC, ISO-8601).
+BOOT_AT: str = datetime.now(timezone.utc).isoformat()
+
+#: OS process id — separates forked workers that share a boot timestamp.
+PID: int = os.getpid()
+
+
+def identity() -> dict[str, object]:
+    """The non-secret process identity block embedded in diagnostics."""
+    return {"instance": INSTANCE_ID, "boot_at": BOOT_AT, "pid": PID}