feat: failure_class + retryable on failed verifications; typed 503 with honoured Retry-After

CHANGELOG.md

@@ -4,6 +4,55 @@ All notable changes to this SDK are documented here. Format follows
 [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); versioning follows
 [SemVer](https://semver.org/).
 
+## [2.8.0] - 2026-08-21
+
+The server now says WHY a verification failed and states honest waits when it
+is overloaded; the SDK types both.
+
+### Added
+- **`failure_class` + `retryable` on failed verifications.** `TaskStatus`, the
+  `VerificationFailed` webhook event, and `LenzPipelineError` all carry
+  `failure_class` (closed set: `upstream_unavailable` | `insufficient_evidence`
+  | `invalid_input` | `cancelled` | `internal`) and `retryable` (true iff
+  `upstream_unavailable` — resubmitting the same claim is the right move).
+  Older servers omit both; the fields default rather than break. The closed
+  set is also exported as the `FailureClass` `Literal` alias
+  (`from lenz_io import FailureClass`) for exhaustive matching — the model
+  field itself stays `str` so a class the server adds later can't turn into a
+  `ValidationError`.
+- **`LenzUpstreamUnavailableError`** — a `LenzAPIError` subclass for 503s with
+  `code` `upstream_unavailable` (model/search providers exhausted; the request
+  was not charged) or `capacity` (submission shed at the door; nothing was
+  accepted). Carries `retry_after`.
+
+### Changed
+- **A 503 that Lenz itself typed — body `code` `upstream_unavailable` or
+  `capacity` — and that asks for more than 60s now raises immediately**, as
+  `LenzUpstreamUnavailableError` carrying the true `retry_after`, instead of
+  silently burning the 1s/2s/4s backoff ladder against a server that asked
+  for 90-120s. That is the same rule 429 has always had. The decision is
+  gated on the body code, not on the status number:
+  - typed 503, stated wait ≤ 60s → still slept through and retried (unchanged);
+  - **untyped 503** — an ordinary proxy / load-balancer / maintenance
+    response with no Lenz `code` — → **backoff ladder, exactly as before**,
+    however long a `Retry-After` it states;
+  - every other 5xx → backoff ladder, unchanged.
+
+  If you relied on long-stated-wait typed 503s being retried blindly, catch
+  `LenzUpstreamUnavailableError` (existing `except LenzAPIError` handlers
+  keep catching it).
+- The stated wait is now also read from the 503 body's `retry_after` key
+  (previously only the `Retry-After` header and the 429 body's
+  `reset_in_seconds`), so a proxy that strips headers can't demote an honest
+  wait to blind backoff.
+
+### Fixed
+- `LenzPipelineError.retryable` now coerces a non-boolean server value to
+  `None` instead of passing it through (parity with the Node SDK).
+- Contract fixtures refreshed to the live failed-status body; added the
+  `verification.failed` webhook payload and both 503 envelopes (shared
+  byte-identically with the Node SDK, as ever).
+
 ## [2.7.1] - 2026-08-15
 
 ### Fixed

README.md

@@ -189,7 +189,7 @@ Every claim-shaped response shares these fields at top level:
 ### Webhooks
 
 ```python
-from lenz_io import LenzWebhooks, VerificationCompleted, VerificationNeedsInput
+from lenz_io import LenzWebhooks, VerificationCompleted, VerificationFailed, VerificationNeedsInput
 
 webhooks = LenzWebhooks(secret="whsec_...")
 
@@ -201,6 +201,13 @@ if isinstance(event, VerificationCompleted):
 elif isinstance(event, VerificationNeedsInput):
     tid, ni = event.task_id, event.needs_input
     ...
+elif isinstance(event, VerificationFailed):
+    # event.error is WHERE the pipeline stopped; event.failure_class is WHY
+    # (closed set) and event.retryable tells you what to do about it.
+    if event.retryable:
+        resubmit_later(event.task_id)  # transient provider outage
+    else:
+        log_permanent_failure(event.task_id, event.error)
 ```
 
 If you're on Python 3.10+ a `match` statement reads even cleaner — events are
@@ -223,6 +230,7 @@ from lenz_io import (
     LenzAuthError,
     LenzQuotaExceededError,
     LenzRateLimitError,
+    LenzUpstreamUnavailableError,
     LenzValidationError,
 )
 
@@ -248,8 +256,21 @@ except LenzRateLimitError as exc:
 except LenzValidationError as exc:
     for field_err in exc.errors:
         print(field_err["loc"], field_err["msg"])
+except LenzUpstreamUnavailableError as exc:
+    # HTTP 503, code "upstream_unavailable" (model/search providers
+    # exhausted) or "capacity" (submissions shed at the door). Nothing was
+    # charged. Waits up to 60s are already slept through by the automatic
+    # retry ladder; reaching here means the server stated a longer one.
+    schedule_retry_in(exc.retry_after)  # typically 90-120s
 ```
 
+A failed *verification* (as opposed to a failed HTTP call) raises
+`LenzPipelineError` from `verify_and_wait` / `wait`. Since 2.8.0 it carries
+`failure_class` (closed set: `upstream_unavailable` | `insufficient_evidence`
+| `invalid_input` | `cancelled` | `internal`) and `retryable` — `True` means
+a transient provider-side exhaustion where resubmitting the same claim is the
+right move; older servers leave it `None`.
+
 `LenzQuotaExceededError` is a **sibling** of `LenzAuthError`, not a subclass —
 "fix your key" and "top up your account" are different actions. So if you were
 catching `LenzAuthError` to handle an empty balance, that branch stops firing;

src/lenz_io/__init__.py

@@ -46,6 +46,7 @@
     LenzQuotaExceededError,
     LenzRateLimitError,
     LenzTimeoutError,
+    LenzUpstreamUnavailableError,
     LenzValidationError,
     LenzWebhookSignatureError,
 )
@@ -64,6 +65,7 @@
     EntityRef,
     ExtractedClaims,
     ExtractedEntity,
+    FailureClass,
     LibraryItem,
     LibraryList,
     RelatedVerifications,
@@ -105,6 +107,7 @@
     "EntityRef",
     "ExtractedClaims",
     "ExtractedEntity",
+    "FailureClass",
     "Lenz",
     "LenzAPIError",
     "LenzAuthError",
@@ -114,6 +117,7 @@
     "LenzQuotaExceededError",
     "LenzRateLimitError",
     "LenzTimeoutError",
+    "LenzUpstreamUnavailableError",
     "LenzValidationError",
     "LenzWebhookSignatureError",
     "LenzWebhooks",

src/lenz_io/client.py

@@ -70,6 +70,7 @@
 from . import __version__
 from .errors import (
     MAX_RETRY_AFTER_SLEEP,
+    UPSTREAM_503_CODES,
     LenzAPIError,
     LenzError,
     LenzNeedsInputError,
@@ -648,13 +649,21 @@ def _verification_from_terminal(self, status: TaskStatus, task_id: str) -> Verif
         # failed. Server sends the diagnostic under ``error``; fall back to the
         # legacy fields for resilience.
         detail = status.error or status.failure_detail or status.failure_reason or "unknown"
+        if status.retryable:
+            fix = "Transient provider outage — retry the same request after a short wait."
+        else:
+            fix = "Retry with a different claim, or check status.error for the diagnostic."
         raise LenzPipelineError(
             message=f"Pipeline failed: {detail}",
             cause=detail,
-            fix="Retry with a different claim, or check status.error for the diagnostic.",
+            fix=fix,
             doc_url="https://lenz.io/docs/errors",
             task_id=task_id,
             failure_reason=status.failure_reason,
+            failure_class=status.failure_class,
+            # Coerce like the Node SDK: only a real boolean is a retry signal;
+            # anything else (a stringy "true", a future enum) reads as unknown.
+            retryable=status.retryable if isinstance(status.retryable, bool) else None,
         )
 
     # ── account ──
@@ -802,22 +811,32 @@ def _request(
             # Error path. Retry on 5xx and 429; otherwise raise immediately.
             #
             # A stated wait is honored only up to MAX_RETRY_AFTER_SLEEP. Past
-            # that the two statuses part ways, because the right answer differs:
+            # that, whether we abort or keep retrying is decided by the typed
+            # body ``code`` — NOT by the status number:
             #
             #  * 429 — raise. The /extract daily cap sends
             #    seconds-until-UTC-midnight, so sleeping it blocks the call for
             #    most of a day, three times over. The caller gets the true
             #    retry_after and can schedule the work.
-            #  * 5xx — keep retrying on our own backoff. The server is down,
-            #    not rate-limiting us; a maintenance-window Retry-After of an
-            #    hour shouldn't become an hour-long sleep, but it also
-            #    shouldn't abort a request our backoff might still satisfy.
+            #  * 503 carrying a Lenz code in UPSTREAM_503_CODES
+            #    (``upstream_unavailable`` / ``capacity``) — raise, same
+            #    reasoning. These are the server's own shed/exhaustion
+            #    responses; they state an honest 90-120s and burning the
+            #    1/2/4s ladder against them is the opposite of what the header
+            #    asks (map_response_to_error types them
+            #    LenzUpstreamUnavailableError, carrying the true retry_after).
+            #  * every other 5xx, including an UNTYPED 503 — keep retrying on
+            #    our own backoff. A Cloud Run / CDN / load-balancer
+            #    maintenance-or-overload 503 states a long wait and carries no
+            #    Lenz code; the server is down, not pacing us, so an hour-long
+            #    Retry-After must become backoff — not an hour-long sleep, and
+            #    not an abort of a request our ladder might still satisfy.
             if attempt < self._max_retries and (response.status_code >= 500 or response.status_code == 429):
                 stated = _stated_retry_after(response)
                 if stated is not None and stated <= MAX_RETRY_AFTER_SLEEP:
                     time.sleep(stated)
                     continue
-                if stated is None or response.status_code >= 500:
+                if stated is None or not _aborts_on_long_stated_wait(response):
                     time.sleep(_retry_sleep(attempt))
                     continue
 
@@ -837,9 +856,11 @@ def _stated_retry_after(response: httpx.Response) -> int | None:
     """Seconds the server says to wait, or None if it didn't say.
 
     Reads the ``Retry-After`` header first, then the body's
-    ``reset_in_seconds``. Returns None (rather than 0) on an absent or
-    unparseable value so the caller can tell "server stated no wait" apart
-    from "server said wait 0 seconds" and fall back to its own backoff.
+    ``reset_in_seconds`` (429 shapes), then the body's ``retry_after``
+    (the 503 shapes carry the wait under that key). Returns None (rather
+    than 0) on an absent or unparseable value so the caller can tell
+    "server stated no wait" apart from "server said wait 0 seconds" and
+    fall back to its own backoff.
     """
     raw = response.headers.get("Retry-After")
     if raw is None or str(raw).strip() == "":
@@ -851,6 +872,8 @@ def _stated_retry_after(response: httpx.Response) -> int | None:
         if not isinstance(body, dict):
             return None
         raw = body.get("reset_in_seconds")
+        if raw is None or str(raw).strip() == "":
+            raw = body.get("retry_after")
     if raw is None or str(raw).strip() == "":
         return None
     try:
@@ -862,6 +885,35 @@ def _stated_retry_after(response: httpx.Response) -> int | None:
         return None
 
 
+def _body_error_code(response: httpx.Response) -> str:
+    """The server's machine-readable ``code`` from the body, or ``""``.
+
+    Reads it exactly the way ``map_response_to_error`` does — string-typed
+    only, so a malformed ``code: 42`` reads as ``""`` rather than ``"42"``
+    and nothing branches on a value the server never meant as a code.
+    """
+    try:
+        body = response.json()
+    except Exception:
+        return ""
+    if not isinstance(body, dict):
+        return ""
+    code = body.get("code")
+    return code if isinstance(code, str) else ""
+
+
+def _aborts_on_long_stated_wait(response: httpx.Response) -> bool:
+    """Whether a stated wait past the cap should abort instead of back off.
+
+    True for 429 (always) and for a 503 the server typed as its own
+    shed/exhaustion response. An untyped 503 — the ordinary proxy /
+    maintenance shape — is deliberately False: it keeps the ladder.
+    """
+    if response.status_code == 429:
+        return True
+    return response.status_code == 503 and _body_error_code(response) in UPSTREAM_503_CODES
+
+
 def _retry_sleep(attempt: int) -> float:
     if attempt < len(RETRY_BACKOFF):
         return RETRY_BACKOFF[attempt]

src/lenz_io/errors.py

@@ -190,6 +190,24 @@ class LenzAPIError(LenzError):
     """500 / 502 / 503 / 504 / catch-all for unexpected server errors."""
 
 
+class LenzUpstreamUnavailableError(LenzAPIError):
+    """503 with ``code`` ``upstream_unavailable`` or ``capacity``.
+
+    The server is telling you the truth about a *transient* condition: its
+    model/search providers are exhausted (``upstream_unavailable`` — nothing
+    was charged, the same request succeeds once they recover) or the pipeline
+    is at capacity (``capacity`` — nothing was accepted or charged). Retry the
+    SAME request after ``retry_after`` seconds.
+
+    Subclasses :class:`LenzAPIError`, so existing ``except LenzAPIError``
+    handlers keep catching it. Waits up to ``MAX_RETRY_AFTER_SLEEP`` are
+    already slept through by the automatic retry ladder — seeing this raised
+    means the stated wait was longer, and ``retry_after`` carries it.
+    """
+
+    retry_after: int | None = None
+
+
 class LenzTimeoutError(LenzError):
     """``verify_and_wait`` exceeded the configured timeout.
 
@@ -212,10 +230,19 @@ class LenzNeedsInputError(LenzError):
 
 
 class LenzPipelineError(LenzError):
-    """``verify_and_wait`` saw a terminal ``failed`` state from the pipeline."""
+    """``verify_and_wait`` saw a terminal ``failed`` state from the pipeline.
+
+    ``failure_class`` says WHY (closed set: ``upstream_unavailable`` |
+    ``insufficient_evidence`` | ``invalid_input`` | ``cancelled`` |
+    ``internal``); ``retryable`` is the derived signal — ``True`` means a
+    transient provider-side exhaustion where resubmitting the same claim is
+    the right move. ``None`` when an older server didn't say.
+    """
 
     task_id: str = ""
     failure_reason: str = ""
+    failure_class: str = ""
+    retryable: bool | None = None
 
 
 class LenzWebhookSignatureError(LenzError):
@@ -246,6 +273,16 @@ class LenzWebhookSignatureError(LenzError):
 # immediately with the true retry_after so the caller can schedule the work.
 MAX_RETRY_AFTER_SLEEP = 60
 
+# The body ``code`` values the server sends on a 503 it produced deliberately:
+# providers exhausted mid-pipeline, or a submission shed at the door. Both map
+# to LenzUpstreamUnavailableError and both state an honest wait.
+#
+# The retry ladder in ``client.py`` keys its immediate-abort decision on THIS,
+# not on the status number: an ordinary Cloud Run / CDN / load-balancer 503
+# carries no Lenz code, states a maintenance-window wait, and must keep being
+# retried exactly as it was before 2.8.0.
+UPSTREAM_503_CODES = ("upstream_unavailable", "capacity")
+
 _STATUS_MAP: dict[int, tuple[type[LenzError], str, str]] = {
     401: (
         LenzAuthError,
@@ -309,6 +346,12 @@ def map_response_to_error(
 
     if status_code in _STATUS_MAP:
         cls, default_msg, doc_url = _STATUS_MAP[status_code]
+    elif status_code == 503 and code in UPSTREAM_503_CODES:
+        cls, default_msg, doc_url = (
+            LenzUpstreamUnavailableError,
+            "Service temporarily unavailable",
+            f"{_DOCS_BASE}/errors#unavailable",
+        )
     elif 500 <= status_code < 600:
         cls, default_msg, doc_url = LenzAPIError, "Server error", f"{_DOCS_BASE}/errors"
     else:
@@ -328,6 +371,14 @@ def map_response_to_error(
 
     # Class-specific enrichment from the response body. Each is set on the
     # instance so callers can access via the documented attribute name.
+    if isinstance(err, LenzUpstreamUnavailableError):
+        # Body ``retry_after`` first (both 503 shapes carry it), header as
+        # the fallback for any proxy that strips the body.
+        stated = _opt_int(parsed.get("retry_after"))
+        if stated is None:
+            stated = _opt_int(headers.get("Retry-After") or headers.get("retry-after"))
+        err.retry_after = stated
+
     if isinstance(err, LenzQuotaExceededError):
         # String-typed only. `str(...)` on a malformed dict would render
         # "{'a': 1}" and friendly_text would show that to a user as a URL.