feat(api): make batch traffic yield to interactive users (STIT-644) - #235
Draft
AlexAxthelm wants to merge 10 commits into
Draft
feat(api): make batch traffic yield to interactive users (STIT-644)#235AlexAxthelm wants to merge 10 commits into
AlexAxthelm wants to merge 10 commits into
Conversation
Adds BatchYieldMiddleware: requests tagged X-Stitch-Traffic-Class: batch wait while interactive traffic is in flight or recently finished, then are admitted anyway after max_wait (never shed, so it cannot fail a caller). Pure ASGI and registered innermost, so a deferred request resolves no dependencies -- no token verify, no DB session. Health paths skip both the wait and the activity accounting. Off by default; inert in prod.
The linkage pass is batch work, so it volunteers to yield to interactive users via the API's batch-yield gate. Composed over the existing bearer token headers provider; no client-package change needed. The readiness probe opts out (tag_as_batch=False): it reports whether the API is reachable, so it must not be throttled by the interactive traffic it is meant to be independent of.
Every authenticated request paid an RSA signature verify, though callers reuse one token for a whole browser session or linkage run. Caches successful validations keyed on a SHA-256 digest (not the token itself), each entry carrying the token's own exp so a hit cannot extend its life. Failures are never cached; bounded at 1024 entries, LRU. Pool limits are now stated rather than inherited. Same values as SQLAlchemy's defaults, so no behavior change -- the point is that the real ceiling is visible: 15 connections per process, checked out twice per authenticated request.
CD summary
|
| service | url | fqdn |
|---|---|---|
| api | open | pr-0235-api.purplegrass-c07d0a94.westus2.azurecontainerapps.io |
| entity-linkage | open | pr-0235-el.purplegrass-c07d0a94.westus2.azurecontainerapps.io |
| frontend | https://witty-mushroom-017a3dc1e-235.westus2.1.azurestaticapps.net | |
| stitch-llm | open | pr-0235-llm.purplegrass-c07d0a94.westus2.azurecontainerapps.io |
Database (1)
| db_name | postgres_host | postgres_port | postgres_db |
|---|---|---|---|
| pr_0235 | stitch-dev.postgres.database.azure.com |
5432 |
pr_0235 |
Jobs (2)
| job | image | postgres_db | api_url | auth_mode |
|---|---|---|---|---|
| db-migrations | ghcr.io/rmi/stitch-api:pr-0235@sha256:2f394909da45b7e0ced609760738bcb64f27964b81bfb835c741beeb2e5d3040 |
pr_0235 |
||
| seed | ghcr.io/rmi/stitch-seed:pr-0235@sha256:8cf340f0a3476c6aa850944e72cc5573f9364d344364325ca68026855d332c99 |
https://pr-0235-api.purplegrass-c07d0a94.westus2.azurecontainerapps.io/api/v1 |
stitch-client-bearer-token |
Images (4)
| build_time | commit_time | git_sha | image | image_digest |
|---|---|---|---|---|
| 2026-08-13T12:12:08Z | 2026-08-13T12:11:53Z | 40e0629 | ghcr.io/rmi/stitch-api:pr-0235 |
ghcr.io/rmi/stitch-api:pr-0235@sha256:2f394909da45b7e0ced609760738bcb64f27964b81bfb835c741beeb2e5d3040 |
| 2026-08-13T12:12:10Z | 2026-08-13T12:11:53Z | 40e0629 | ghcr.io/rmi/stitch-entity-linkage:pr-0235 |
ghcr.io/rmi/stitch-entity-linkage:pr-0235@sha256:f42a691ff3ad03173622edb6ce7c7d7472af29c4b07fa50f7dd96a49644bfa4f |
| 2026-08-13T12:12:13Z | 2026-08-13T12:11:53Z | 40e0629 | ghcr.io/rmi/stitch-seed:pr-0235 |
ghcr.io/rmi/stitch-seed:pr-0235@sha256:8cf340f0a3476c6aa850944e72cc5573f9364d344364325ca68026855d332c99 |
| 2026-08-13T12:12:11Z | 2026-08-13T12:11:53Z | 40e0629 | ghcr.io/rmi/stitch-stitch-llm:pr-0235 |
ghcr.io/rmi/stitch-stitch-llm:pr-0235@sha256:185dbab7199dc66c5107dd5d4126dd96b17281f9824622f5fb46d93d11f208b4 |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces cooperative admission control in the API so batch workloads (entity-linkage) yield to interactive traffic on shared dev stacks, while keeping production behavior unchanged by default. It also reduces per-request auth CPU cost via memoized token-claims validation and makes DB pool ceilings explicit for operational clarity.
Changes:
- Added an ASGI-level “batch yield” admission gate (dev-only, opt-in via env/settings) driven by
X-Stitch-Traffic-Class: batch. - Updated entity-linkage to tag its API calls as batch by default, while ensuring health/readiness probes are not batch-tagged.
- Added a bounded, exp-aware validated-claims cache to avoid repeated RSA verification on reused bearer tokens; declared explicit SQLAlchemy pool limits.
Reviewed changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| PLAN.md | Design/rollout plan documenting rationale, constraints, risks, and verification steps. |
| env.example | Documents BATCH_YIELD_* env vars for enabling/tuning the dev-only gate. |
| deployments/entity-linkage/src/stitch/entity_linkage/client.py | Adds traffic-class header tagging for batch requests and a tag_as_batch switch for probes. |
| deployments/entity-linkage/src/stitch/entity_linkage/routers/health.py | Ensures downstream API reachability check is not batch-tagged (won’t be delayed by the gate). |
| deployments/entity-linkage/tests/test_client.py | Adds coverage to ensure batch tagging is present when expected and absent for probes. |
| deployments/api/src/stitch/api/admission.py | New pure-ASGI middleware implementing the batch-yield admission gate + structured logging. |
| deployments/api/src/stitch/api/middleware.py | Registers the gate only when enabled and never activates it in prod (warns if misconfigured). |
| deployments/api/src/stitch/api/settings.py | Adds batch_yield_* settings with bounds and sensible defaults. |
| deployments/api/src/stitch/api/auth.py | Adds bounded validated-claims cache keyed by token digest and respecting token exp. |
| deployments/api/tests/test_admission.py | Comprehensive deterministic tests for gate classification, exemptions, timing semantics, and registration order. |
| deployments/api/tests/test_auth_unit.py | Adds unit tests covering cache correctness, expiry behavior, and bounding. |
| deployments/api/src/stitch/api/db/config.py | Makes pool limits explicit (matching SQLAlchemy defaults) and documents implications. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
AlexAxthelm
changed the title
CD summary
|
| service | url | fqdn |
|---|---|---|
| api | open | pr-0235-api.purplegrass-c07d0a94.westus2.azurecontainerapps.io |
| entity-linkage | open | pr-0235-el.purplegrass-c07d0a94.westus2.azurecontainerapps.io |
| frontend | https://witty-mushroom-017a3dc1e-235.westus2.1.azurestaticapps.net | |
| stitch-llm | open | pr-0235-llm.purplegrass-c07d0a94.westus2.azurecontainerapps.io |
Database (1)
| db_name | postgres_host | postgres_port | postgres_db |
|---|---|---|---|
| pr_0235 | stitch-dev.postgres.database.azure.com |
5432 |
pr_0235 |
Jobs (1)
| job | image | postgres_db |
|---|---|---|
| db-migrations | ghcr.io/rmi/stitch-api:pr-0235@sha256:6c84e07010030ab987fe2b48616fcfa324be83693633c519c8ce988d12193d98 |
pr_0235 |
Images (4)
| build_time | commit_time | git_sha | image | image_digest |
|---|---|---|---|---|
| 2026-08-13T12:45:42Z | 2026-08-13T12:45:23Z | b835ad3 | ghcr.io/rmi/stitch-api:pr-0235 |
ghcr.io/rmi/stitch-api:pr-0235@sha256:6c84e07010030ab987fe2b48616fcfa324be83693633c519c8ce988d12193d98 |
| 2026-08-13T12:45:47Z | 2026-08-13T12:45:23Z | b835ad3 | ghcr.io/rmi/stitch-entity-linkage:pr-0235 |
ghcr.io/rmi/stitch-entity-linkage:pr-0235@sha256:54b29b23f253f0a8fdba61e7859a89e9a22f282ce28272d8be938333e116e108 |
| 2026-08-13T12:45:42Z | 2026-08-13T12:45:23Z | b835ad3 | ghcr.io/rmi/stitch-seed:pr-0235 |
ghcr.io/rmi/stitch-seed:pr-0235@sha256:6b3528f8c9d331369755fea9d1a2998ccaa1633153d186c38de03a8bdfc2b9f7 |
| 2026-08-13T12:45:42Z | 2026-08-13T12:45:23Z | b835ad3 | ghcr.io/rmi/stitch-stitch-llm:pr-0235 |
ghcr.io/rmi/stitch-stitch-llm:pr-0235@sha256:e1336332b8c2deed4fe1f243d5fbe5f00ad2750daef0730c39815f26ff075422 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
tl;dr: add an optional HTTP header
X-Stitch-Traffic-Class: batchto allow requests to flag themselves as lower priority (They will wait until untagged requests are done before processing)Note this doesn't actually solve the problem of "EL makes a lot of traffic", but it does help control by putting up a yield sign for the batch jobs. This is a low-effort very basic approach, but it doesn't exclude any better options later. We should consider adding to ETL and seed at some point.
Problem
STIT-644 — a linkage run saturates the shared API and leaves too little capacity for a human to use the app (found in the B&L demo).
Entity linkage calls the same public API surface as the frontend. That's a good programming model and this PR keeps it.
Why not a concurrency limit
EL's in-flight HTTP concurrency is already 1:
JobManagerpermits one run andlink_allawaits every call sequentially. A semaphore sized above 1 would never engage.EL saturates through duty cycle, not parallelism — ~3+ requests per resource, back to back, against the most expensive query in the system, with zero think time. So this is a rate control, not a concurrency limit.
What this does
A request tagged
X-Stitch-Traffic-Class: batchwaits while interactive requests are in flight or recently finished, then is admitted anyway aftermax_wait— never a 429, so it cannot fail a linkage run. Untagged requests are interactive, so there is nothing worth spoofing.BaseHTTPMiddleware, whosecall_nextreturns at response headers and would release the in-flight count too early), registered innermost so a deferred request resolves no dependencies — no token verify, no DB session.quiet_ms > 5000starve batch traffic with zero human load.BATCH_YIELD_ENABLED); warns and stays inert in prod.Two per-request cost reductions that help interactive traffic equally:
exp, failures never cached) — removes an RSA verify from every repeat request.Enable locally with
BATCH_YIELD_ENABLED=truein.env.Scope
Not the root cause:
link_allis O(resources × pages) against theq=search, which STIT-619 tracks. This makes the shared server usable meanwhile. The two compose — the gate acts per request at admission, so it still works once STIT-619 gives EL concurrent requests.Deriving traffic class from caller identity rather than a header becomes possible with M2M (STIT-469 / STIT-424); the header stays the mechanism either way.
The entity-linkage container cpu/memory unpin is on a separate branch.
AI assistance
Claude Code was used for codebase exploration, design review, and drafting the implementation and tests.
Verified locally:
make py-test(562 pass),py-lint,py-format-check,py-lock-checkall green.Tests:
deployments/api/tests/test_admission.py(40 tests, 0.2s) covers classification, exemptions, quiet-window behavior via an injected clock, in-flight accounting under handler exceptions and client cancellation, registration order, and prod inertness — plus one deliberate real-clock test so a mis-wired default clock/sleep can't slip through. Claims-cache behavior intest_auth_unit.py.Not verified: the end-to-end docker comparison (baseline vs. under-linkage latency via
tools/analyze_logs.py --group-by scenario). Worth doing against the preview env.