diff --git a/docs/perspectives/README.md b/docs/perspectives/README.md index 4a6959b..e3ae0d8 100644 --- a/docs/perspectives/README.md +++ b/docs/perspectives/README.md @@ -166,6 +166,156 @@ followed: expressible rules, and an eval-scorer seam for LLM-backed capabilities. +## Round 8 — structure becomes vocabulary + +The round-7 asks shipped as a release (`stats[:roots]`/`stats[:leaves]`, +journal `duration_samples` with `duration_percentile(desc, pct, last:)`, +and `x-agentic-rules` emission in `to_json_schema`), and ten more +experiments followed: + +| # | Persona | Built on the round-8 release | Run it | Field notes | +|---|---------|------------------------------|--------|-------------| +| 1 | Matz | Plan forest — the graph drawn as trees, depth as altitude | `examples/plan_forest.rb` | [round-8/01-matz.md](round-8/01-matz.md) | +| 2 | DHH | Hill chart — where the work *is*, from lifecycle hooks | `examples/hill_chart.rb` | [round-8/02-dhh.md](round-8/02-dhh.md) | +| 3 | Aaron Patterson | Variance detective — flaky vs slow, settled by percentiles | `examples/variance_detective.rb` | [round-8/03-tenderlove.md](round-8/03-tenderlove.md) | +| 4 | Xavier Noria | Plan merge — three-way merge with conflicts at seam altitude | `examples/plan_merge.rb` | [round-8/04-fxn.md](round-8/04-fxn.md) | +| 5 | Samuel Williams | Adaptive throttle — AIMD finds the capacity nobody documented | `examples/adaptive_throttle.rb` | [round-8/05-ioquatix.md](round-8/05-ioquatix.md) | +| 6 | Jeremy Evans | Journal audit — five invariants; a tampered journal named precisely | `examples/journal_audit.rb` | [round-8/06-jeremyevans.md](round-8/06-jeremyevans.md) | +| 7 | Piotr Solnica | Contract semver — breaking-or-compatible computed, bump advised | `examples/contract_semver.rb` | [round-8/07-solnic.md](round-8/07-solnic.md) | +| 8 | Mike Perham | Dead letter office — requeue, parked, recovered, by last word | `examples/dead_letter_office.rb` | [round-8/08-mperham.md](round-8/08-mperham.md) | +| 9 | Sandi Metz | Graph to specs — structural roles dictate the test plan | `examples/graph_to_specs.rb` | [round-8/09-sandimetz.md](round-8/09-sandimetz.md) | +| 10 | Andrew Kane | Eval scorers — four ways to say "good enough", one seam | `examples/eval_scorers.rb` | [round-8/10-ankane.md](round-8/10-ankane.md) | + +### What round 8 surfaced + +1. **Structure became vocabulary**: `roots`/`leaves`/`depth` landed and + were immediately spent three ways — a drawing (forest), a test plan + (graph-to-specs), and merge conflicts named at seam altitude. Metadata + that keeps buying unplanned tools is metadata shaped right. +2. **Durations became distributions**: `duration_samples` turned point + readings into percentiles, and two tools acted on them — the variance + detective separates flaky from slow (p90/p50 ratio), and the adaptive + throttle steers concurrency by p50 drift instead of vibes. +3. **Signal-to-noise as a design goal**: the dead letter office triages + by *most recent* attempt (no paging for ghosts), and the eval scorers + flag exactly one real failure where exact-match flags two. Both argue + the same point: a report is only as good as what its failures mean. +4. **The declarations' blind spot held**: Piotr's semver advisor and + Jeremy's audit both stop at callable rules — predicates stay opaque + to every static tool. Structured rules narrow the gap; they don't + close it. +5. **Next asks**: `RateLimit#resize(n)` so the adaptive throttle can + steer the real limiter instead of simulating one, and journaling + `retryable:` at write time from `failure.retryable?` so triage + survives taxonomy renames. (`eval_scorers.rb` joins the + exit-1-by-design set.) + +## Round 9 — the operations round + +The round-8 asks shipped as a release (`RateLimit#resize(n)` — live +ceiling changes, growing wakes waiters, shrinking drains — and the +journal recording `retryable:` on `task_failed` at write time from the +failure's own verdict), the two examples that asked were modernized +onto them, and ten more experiments followed: + +| # | Persona | Built on the round-9 release | Run it | Field notes | +|---|---------|------------------------------|--------|-------------| +| 1 | Matz | Failure weather — retryable is weather, non-retryable is climate | `examples/failure_weather.rb` | [round-9/01-matz.md](round-9/01-matz.md) | +| 2 | DHH | Traffic dial — a canary rollout as one resized limiter | `examples/traffic_dial.rb` | [round-9/02-dhh.md](round-9/02-dhh.md) | +| 3 | Aaron Patterson | Throughput knee — the ceiling sweep with two honest clocks | `examples/throughput_knee.rb` | [round-9/03-tenderlove.md](round-9/03-tenderlove.md) | +| 4 | Xavier Noria | Graph invariants — seven promises of the reflection API, proved | `examples/graph_invariants.rb` | [round-9/04-fxn.md](round-9/04-fxn.md) | +| 5 | Samuel Williams | Fair share — tenant-fairness composed, shares rebalanced live | `examples/fair_share.rb` | [round-9/05-ioquatix.md](round-9/05-ioquatix.md) | +| 6 | Jeremy Evans | Resize torture — shrink drains, grow wakes, ceilings bind | `examples/resize_torture.rb` | [round-9/06-jeremyevans.md](round-9/06-jeremyevans.md) | +| 7 | Piotr Solnica | Contract fixtures — examples derived from declarations, proved | `examples/contract_fixtures.rb` | [round-9/07-solnic.md](round-9/07-solnic.md) | +| 8 | Mike Perham | Circuit breaker — three strikes for 503s, one for a revoked key | `examples/circuit_breaker.rb` | [round-9/08-mperham.md](round-9/08-mperham.md) | +| 9 | Sandi Metz | Duck agents — five shapes through one seam, one tiny decorator | `examples/duck_agents.rb` | [round-9/09-sandimetz.md](round-9/09-sandimetz.md) | +| 10 | Andrew Kane | Impl shootout — accuracy AND latency on one table | `examples/impl_shootout.rb` | [round-9/10-ankane.md](round-9/10-ankane.md) | + +### What round 9 surfaced + +1. **resize turned limits into policy objects**: five tools steer one + live limiter — the rollout dial, the ceiling sweep, tenant share + rebalancing, the torture certificate, and the modernized AIMD + throttle. The topology of a limiter graph stays fixed; only the + numbers move at runtime, which is the property that makes it safe. +2. **The write-time verdict became a decision input**: the weather + report (wait vs dig a well), the circuit breaker (three strikes vs + instant trip), and the modernized dead letter office all *act* on + `retryable:` instead of reconstructing it — the error's testimony, + recorded when fresh, drives policy later. +3. **The tools kept correcting their authors** (fifth consecutive + round): Samuel's one-worker tenant couldn't starve, Aaron's + "throughput goes flat" was actually a fall, Xavier's depth + invariant was ill-posed on cycles, Jeremy's harness read its clock + before setting it, Mike's breaker read the wrong journal event, and + Kane's challenger lost a case to a missing stem. Every one was + caught by the example's own output before a user saw it. +4. **Fairness needs unmet demand to be visible**: a FIFO door is fair + to requests, not tenants — starvation only appears when a tenant's + demand exceeds its receipts, which is why quiet outages stay quiet. +5. **Next asks**: relation-typed structured rules (`sum_lte:`, + `requires:`, `mutually_exclusive:`) so generators can satisfy and + advisors can diff the declarable majority of cross-field rules + (Piotr); and a breaker-friendly convention for `retryable: nil` — + "no opinion" should mean retry-with-suspicion, not hopeless (Mike). + +## Round 10 — predicates become data + +The round-9 asks shipped as a release (`Agentic::RelationRules` — +`sum_lte`/`requires`/`mutually_exclusive` declared as data, enforced +by the validator with derived messages, projected into real draft-07 +keywords, and carried in `x-agentic-rules`; plus the retryable-nil +convention on `TaskFailure`: `hopeless?` / `possibly_transient?`), +the two asking examples were modernized, and ten more experiments +followed: + +| # | Persona | Built on the round-10 release | Run it | Field notes | +|---|---------|-------------------------------|--------|-------------| +| 1 | Matz | Polite form — every declaration read aloud as a question | `examples/polite_form.rb` | [round-10/01-matz.md](round-10/01-matz.md) | +| 2 | DHH | One-file API — schema, 422s, and 201s derived from one declaration | `examples/one_file_api.rb` | [round-10/02-dhh.md](round-10/02-dhh.md) | +| 3 | Aaron Patterson | Contract overhead — validation priced against the call it guards | `examples/contract_overhead.rb` | [round-10/03-tenderlove.md](round-10/03-tenderlove.md) | +| 4 | Xavier Noria | Projection agreement — both renderings of the law, proved; the nil frontier mapped | `examples/projection_agreement.rb` | [round-10/04-fxn.md](round-10/04-fxn.md) | +| 5 | Samuel Williams | Cancel drill — task cancel is prompt; plan cancel bills you anyway | `examples/cancel_drill.rb` | [round-10/05-ioquatix.md](round-10/05-ioquatix.md) | +| 6 | Jeremy Evans | Relation prober — 13 probes pass; one step off the road draws blood | `examples/relation_prober.rb` | [round-10/06-jeremyevans.md](round-10/06-jeremyevans.md) | +| 7 | Piotr Solnica | Relation diff — the rules join semver; opacity becomes opt-in | `examples/relation_diff.rb` | [round-10/07-solnic.md](round-10/07-solnic.md) | +| 8 | Mike Perham | Retry budget — one fleet-wide wallet; 45 doomed calls become 17 | `examples/retry_budget.rb` | [round-10/08-mperham.md](round-10/08-mperham.md) | +| 9 | Sandi Metz | Rule shapes — one policy, three representations, four consumers | `examples/rule_shapes.rb` | [round-10/09-sandimetz.md](round-10/09-sandimetz.md) | +| 10 | Andrew Kane | Batch import — a reject file with line, field, and rule, at 162us/row | `examples/batch_import.rb` | [round-10/10-ankane.md](round-10/10-ankane.md) | + +### What round 10 surfaced + +1. **Predicates as data compounded immediately**: within one round, + relation rules were asked as questions (polite form), served as + draft-07 keywords (one-file API), satisfied by the generator, + diffed for semver, audited for consumer count, and used to explain + 118 CSV rejections. Six consumers for a feature shipped that + morning — the strongest version yet of "metadata keeps buying + unplanned tools." +2. **Two real defects found by drills**: `cancel_plan` under a joined + reactor is bookkeeping-only — every agent runs and bills while the + status says canceled (Samuel); and a relation rule over an + undeclared field escapes as raw TypeError instead of + ValidationError, turning 422 paths into 500 paths (Jeremy, whose + prober exits 1 by design as the acceptance test). +3. **Agreement-for-different-reasons is a named hazard**: typed + fields guard the nil frontier so both renderings reject + `{express: nil}` — for unrelated reasons; relax the type and the + renderings diverge. Verdict-only tests would call that a pass + (Xavier). +4. **The meter settled the validation debate**: the largest contract + costs 0.14ms against the 800ms call it guards, and rejection + costs 12x the happy path — both numbers now on the table (Aaron, + Kane concurring at 162us/row with rejects included). +5. **Next asks**: make `cancel_plan` stop the scheduler and in-flight + fibers (the cancel drill is the acceptance test); relation rules + must type-check their fields at declaration time or wrap + evaluation failures in ValidationError (the relation prober is + the acceptance test); `RateLimit#try_acquire` for non-blocking + admission so retry budgets can be RateLimits; and align or + document presence semantics (Ruby nil vs JSON null) across the + projection boundary. (`relation_prober.rb` joins the + exit-1-by-design set.) + ### What round 6 surfaced 1. **Plans became artifacts**: narratable (tour), serializable with an diff --git a/docs/perspectives/round-10/01-matz.md b/docs/perspectives/round-10/01-matz.md new file mode 100644 index 0000000..15cf015 --- /dev/null +++ b/docs/perspectives/round-10/01-matz.md @@ -0,0 +1,61 @@ +# Round 10 field notes — Matz asks before it's an error + +*Built: `examples/polite_form.rb` — a form assistant that turns the +contract's declarations into questions: requireds become requests, +bounds become gentle corrections, relations become follow-ups.* + +## What I built and why + +An error message is just a question you asked too late. A 422 that +says "express requires customs_code" contains a perfectly good +question — *"since you chose express, may I have your customs +code?"* — wearing armor. I wanted to take the armor off: + +``` +assistant: may I have your mode? (air, sea, road) +assistant: ah - weight must be less than or equal to 5000. shall we adjust it? +assistant: together weight and volume come to 7000, and 6000 is our + limit - could we lower the volume? +assistant: since you chose express, I'll also need your customs_code +assistant: you've given me api_key and oauth_token - I only need one; + which shall we keep? +``` + +Six questions, zero errors shown, and *nothing was written twice*: +every line of the conversation is a declaration read aloud in a +kinder register. `required:` became a request, `max:` a correction, +and — this is the part only possible since this morning — the three +relation rules each became their natural follow-up. `requires` is +"then I'll also need"; `sum_lte` is "could we lower it"; and +`mutually_exclusive` is "which shall we keep?" + +## Why relations made this possible + +Last round the generator could *satisfy* relations silently; this +round the assistant can *discuss* them, and the difference is the +same one: the predicate is data now. A lambda rule could only ever +say pass or fail — you cannot ask a lambda which field to lower, or +what the limit is, or which two things conflict. The relation +declaration carries all three, so the assistant reads the `fields:`, +the `limit:`, and the relation's own shape, and phrases the question +a human clerk would ask. Omotenashi is anticipating the need before +the failure; it turns out anticipation is a data-model feature. + +## Notes + +- The engine is a ten-line loop: validate, catch, convert the first + violation to a question, repeat. Fixed-point politeness. I enjoyed + that convergence is guaranteed by the same property that makes the + validator honest — every question, answered, strictly shrinks the + violation set. +- The `mutually_exclusive` case is the only one where the assistant + *removes* something rather than requesting it, and it asks + permission first. Deleting a user's input without asking is the + form equivalent of clearing their cart. + +## Verdict + +The contract now has two voices — the strict one for machines (422s, +schemas) and this one for people — and both read from the same +declarations, so they can never disagree. Kindness that stays +synchronized with correctness: that is my favorite kind of feature. diff --git a/docs/perspectives/round-10/02-dhh.md b/docs/perspectives/round-10/02-dhh.md new file mode 100644 index 0000000..6228448 --- /dev/null +++ b/docs/perspectives/round-10/02-dhh.md @@ -0,0 +1,60 @@ +# Round 10 field notes — DHH ships the API that isn't there + +*Built: `examples/one_file_api.rb` — a complete endpoint derived from +one capability declaration: schema endpoint, 422s with relation +rules explained, output-guarded 201s.* + +## What I built and why + +Look at a typical API codebase and count the artifacts per endpoint: +a controller, a params validator, a serializer, an OpenAPI YAML that +disagrees with all three, and a test file whose main job is keeping +the other four honest. Five files, one idea. The idea is: *a quote +request has a mode, a weight, and some rules.* + +So say that once, and derive the rest: + +``` +GET /quotes/schema -> 200 (draft-07, 687 bytes) +POST {"mode":"teleport","weight":9000} -> 422 field errors +POST {"weight":4000,"volume":3000} -> 422 "weight + volume + must total at most 6000" +POST {"express":true} -> 422 "express requires + customs_code" +POST (all in order) -> 201 {"price_cents":1800} +``` + +The app — the actual business — is four lines (`create_quote`). The +"API layer" is a case statement that *reads the declaration*: the +422 renderer never mentions a field name, the schema endpoint is one +method call, and `validate_outputs!` guards the response door too, +so the endpoint can't quietly ship a malformed 201 when someone +refactors the pricing. + +## Relations flow to both doors + +The round-10 payoff is that the cross-field laws now reach both +audiences without being written twice. The human at the terminal +gets "express requires customs_code" in the 422 — a sentence, +derived. The client generator gets `"dependencies": {"express": +["customs_code"]}` in the schema — draft-07 a stock validator +enforces client-side, before the request is even sent. Same law, +two renderings, one source. That used to require a platform team +with a style guide; now it's a property of the data model. + +## Notes + +- My first 422 printed each rule violation twice — once flattened + into the `base` field errors, once structured. The renderer now + excludes `base` and keeps the structured form, because a client + that can point at `fields: ["weight", "volume"]` should never have + to parse prose to find out where to put the red border. +- I kept `additionalProperties: true` on display in the schema + rather than hiding unknown-key tolerance. Postel was right and + your API clients are sloppy; design for it in the open. + +## Verdict + +One declaration, three doors: docs, rejection, and response — all +derived, none drifting. The best code in your app is the code that +isn't there, and this endpoint is mostly made of it. diff --git a/docs/perspectives/round-10/03-tenderlove.md b/docs/perspectives/round-10/03-tenderlove.md new file mode 100644 index 0000000..e77fd0f --- /dev/null +++ b/docs/perspectives/round-10/03-tenderlove.md @@ -0,0 +1,66 @@ +# Round 10 field notes — Aaron Patterson reads the meter + +*Built: `examples/contract_overhead.rb` — the validator benchmarked +across contract sizes and rule counts, priced as a fraction of the +LLM call it protects.* + +## What I built and why + +Sooner or later someone says "we skip validation on the hot path, +for performance." That sentence contains a number, and nobody in the +room knows what it is. So: measure. 2,000 validations per row, warm +cache (the first call pays dry-schema compilation — measuring that +would be benchmarking the wrong thing), and the one framing that +matters, which is that **overhead is a fraction**. Everyone quotes +the numerator; the denominator here is an 800ms model round-trip. + +``` +3 keys, no rules 0.0198ms 0.0025% of the call +10 keys, no rules 0.0411ms 0.0051% +10 keys, 5 relations 0.0600ms 0.0075% +30 keys, 15 relations 0.1426ms 0.0178% +rejection, 5 rules broken 0.7316ms (the slow path) +``` + +The whole table rounds to zero. The largest contract I could +pretend was realistic costs a seventh of a millisecond — 0.018% of +the call it guards. Five relation rules add twenty microseconds +over bare keys; relations scale linearly and gently. Skipping +validation "for performance" saves a rounding error and risks +shipping a malformed prompt to a call that *bills you for the +mistake*. That's not an optimization, it's a lottery ticket where +you pay to lose. + +## The slow path is the interesting row + +Rejection costs 0.73ms — 12x the happy path. That's the exception +plus five rule-violation reports being built, and it's the row I'd +watch in a hostile environment: if an attacker can make you *reject* +cheaply-sent garbage at 0.73ms a pop, the validator is your first +line of DoS absorption, not your bottleneck — but it's worth knowing +that failure costs more than success, because capacity planning on +the happy path is how systems fall over on the sad one. + +One measurement note, since benchmarks lie by default: the warm-up +call matters. Cold, the first validation compiles a dry-schema and +costs ~50x the steady state; a naive loop would smear that spike +across the average and report validation as 'slow'. Separate your +one-time costs from your per-call costs or you'll optimize the +wrong one. + +## Notes + +- Relations were the round-10 worry — "predicates as data" sounds + like interpretation overhead. The meter says: 4 microseconds per + relation. Building the lambda from the declaration happens once + per validation, and it's three hash lookups and a closure. Data + won. +- The 800ms denominator is conservative. Against a reasoning-model + call measured in seconds, the fraction gains another zero. + +## Verdict + +"Can we afford to validate?" was never the question — the question +is whether you can afford not to, and now both numbers are on the +table: 0.14ms against an 800ms call that charges for malformed +input. Validate both doors. The meter says you can afford it. diff --git a/docs/perspectives/round-10/04-fxn.md b/docs/perspectives/round-10/04-fxn.md new file mode 100644 index 0000000..3f3820f --- /dev/null +++ b/docs/perspectives/round-10/04-fxn.md @@ -0,0 +1,68 @@ +# Round 10 field notes — Xavier Noria walks to the frontier + +*Built: `examples/projection_agreement.rb` — every presence +combination evaluated against both renderings of the relation rules +(Ruby validator, draft-07 projection), agreement proved point by +point, and the exact frontier where the renderings part ways, +mapped.* + +## What I built and why + +This round the relation rules began rendering twice: the validator +enforces them, and `to_json_schema` projects `requires` into +`dependencies` and `mutually_exclusive` into not-required clauses. +Two renderings of one law is exactly the situation where drift is +born — nothing forces a projection to stay faithful except a proof +that re-runs. So: four fields, sixteen presence combinations, both +evaluators, demand agreement on every point. + +``` +16 combinations, 0 disagreements +``` + +The draft-07 side is evaluated by a four-line interpreter for +exactly the projected keywords — deliberately not a schema library, +because the proof should depend on the spec text, not on another +implementation's opinions of it. + +## The frontier, surveyed precisely + +Sixteen agreements would have been a boring (if load-bearing) +result, so I walked to where I knew the metaphysics differ: Ruby's +relation presence is *given and non-nil*; JSON Schema's +`dependencies` trigger on the property *existing*, null or not. +`{express: nil}` should split them. + +It didn't — and the reason is the finding. For a **typed** field, +nil never reaches the relation check: per-key typing rejects it +first ("must be boolean"), and the schema rejects it too +(dependencies fire). Agreement, but *for different reasons* — the +most dangerous kind of agreement, because it dissolves the moment +someone relaxes a type. I proved that by declaring `express` without +a type: nil sails past per-key checks, the validator's relation +treats it as absent and allows, the schema's dependencies treat null +as present and reject. There it is: the true divergence, exhibited +on the one plane where it exists. + +So the certificate reads, in full: *the projection is faithful on +the nil-free plane; typed fields guard the frontier; untyped fields +plus explicit null is the crack.* Senders should omit keys, never +null them. Filed as the round-11 ask: align presence semantics +across the boundary or document them as officially distinct. + +## Notes + +- This is the survey-map pattern from my round-9 prover again: the + value isn't "it agrees," it's *knowing the exact shape of where it + doesn't*. An unscoped promise is a bug that hasn't picked its + reporter yet; this promise is now scoped to the character. +- Agreement-for-different-reasons deserves its own name in testing + folklore. Both doors said no, one for typing, one for presence — + a test asserting only the verdict would have called that a pass + and learned nothing. + +## Verdict + +Both renderings of the law agree everywhere the law is meant to +apply, and the one crack is mapped, named, and filed. Exit 0 — a +certificate with its own margins drawn in. diff --git a/docs/perspectives/round-10/05-ioquatix.md b/docs/perspectives/round-10/05-ioquatix.md new file mode 100644 index 0000000..4b8d8a8 --- /dev/null +++ b/docs/perspectives/round-10/05-ioquatix.md @@ -0,0 +1,73 @@ +# Round 10 field notes — Samuel Williams runs the cancel drill + +*Built: `examples/cancel_drill.rb` — three measured drills against +the two cancellation paths: surgical task cancel (in-flight and +pending) and plan-wide cancel. One of them fails the drill.* + +## What I built and why + +Structured concurrency makes exactly one non-negotiable promise: +**stop means stop, promptly**. Everything else — nurseries, barriers, +scoped lifetimes — exists to make that promise keepable. So before I +trust a cancel API in anything that bills by the token, I drill it, +and the drill is always the same: don't read the status, read the +*clock* and the *invoice*. + +``` +drill 1 - cancel one in-flight task at 30ms: + job2 began at 32ms on the canceled fiber's lane - not at 100ms +drill 2 - cancel one pending task: + agents actually ran: 5/6 - the canceled job never started, never billed +drill 3 - cancel_plan at 30ms: + status flipped to :canceled by 30ms... then the plan ran 301ms + anyway, 6/6 agents executed, results discarded +``` + +Drills 1 and 2 pass beautifully. `cancel_task` on an in-flight task +stops the fiber mid-sleep — and the proof is the *next job's start +time*: job2 began at 32ms on the freed lane, not at 100ms when the +canceled job would have finished. Canceling a pending task is even +better: it simply never runs. Queued work canceled is money returned. + +## Drill 3 is the finding + +`cancel_plan` flipped every status to `:canceled` within +milliseconds — and then the plan ran its full 300ms with **all six +agents executing**, their results thrown away on arrival. That's the +worst trade available: full cost, zero product. A dashboard would +show a plan canceled at 30ms; the invoice would show six completed +LLM calls; and both would be telling the truth about different +things, which is the most expensive kind of true. + +The mechanism: `cancel_plan` stops `@reactor` — but when +`execute_plan` joins an existing reactor (the composability we built +in round 1!), that handle isn't the private event loop it was +written to be, and stopping it doesn't reach the scheduler or the +in-flight fibers. Meanwhile the pending→canceled bookkeeping doesn't +stop `schedule_dependent_tasks` from starting those very tasks. The +promise breaks precisely at the intersection of two features that +each work alone. That's not a rare shape of bug; it's the *usual* +shape, and it's why you drill. + +Filed as the round-11 ask, with the drill as the acceptance test: +`cancel_plan` must stop the scheduler and the in-flight fibers — +`@barrier.stop` and per-task stops, not a reactor-handle stop — so +that drill 3 reads like drills 1 and 2. + +## Notes + +- Every claim in the output is a measurement: start timestamps prove + lane-freeing, agent-run counters prove billing, wall clocks prove + promptness. Status fields are testimony; clocks are evidence. +- Note drill 1's subtlety: total wall time was 300ms with or without + the freed lane — my first draft "proved" freeing from the total and + the arithmetic didn't hold. Only the third job's start time + discriminates. Sixth consecutive round of the tools correcting + their authors. + +## Verdict + +Task-level cancellation keeps the structured-concurrency promise; +plan-level cancellation currently sells its status cheaper than its +work. The drill is written, the ask is filed, and next round drill 3 +should cost 30 milliseconds instead of 300. diff --git a/docs/perspectives/round-10/06-jeremyevans.md b/docs/perspectives/round-10/06-jeremyevans.md new file mode 100644 index 0000000..825a9f4 --- /dev/null +++ b/docs/perspectives/round-10/06-jeremyevans.md @@ -0,0 +1,78 @@ +# Round 10 field notes — Jeremy Evans probes the new predicates + +*Built: `examples/relation_prober.rb` — thirteen edge probes against +a hand-written oracle for the three relations, then one deliberate +step off the paved road. The last probe draws blood; exit 1 by +design until the edge is filed down.* + +## What I built and why + +Relation-typed rules shipped this morning. New predicates deserve +hostility on day one, because day one is when their semantics are +still cheap to change. The prober asks the boring questions with +edge inputs — zeros, floats, negatives, missing keys, empty strings — +and checks every verdict against an oracle I wrote by hand, not +against the implementation's own opinion of itself: + +``` +sum_lte: exactly at the limit allow (lte means lte) +sum_lte: negative rescues the sum allow (15 + -6 <= 10) +sum_lte: missing field counts as 0 allow (documented, now proven) +requires: three-field chain broken reject +mutually_exclusive: empty string reject ("" is present - presence + is not truthiness) +13 probes, 0 divergences on the paved road +``` + +Two of those rows are the kind of semantic that starts arguments in +code review, which is exactly why they're pinned here: a *negative* +value can rescue a sum (arithmetic doesn't moralize), and an *empty +string* is present (the mutually-exclusive check counts given keys, +not truthy values — give both credentials, even blank ones, and you +are holding two credentials). + +## Off the paved road + +Then the probe that matters. A rule may reference a field the +contract never declared — nothing forbids it, and per-key validation +cannot type-check what isn't declared. So a string sails through to +`sum_lte`'s arithmetic: + +``` +RAW TypeError: "String can't be coerced into Integer" +``` + +A validator has one job: convert bad input into its *own* error +type, every time, so callers can write `rescue ValidationError` and +mean it. Here, sufficiently bad input crashes the validator instead. +Every 422 path guarding this code is silently also a 500 path, and +nobody's rescue clause knows it. This is the fail-open cousin of the +string-`raise` sins from round 1 — the failure isn't hidden, but it +arrives wearing the wrong uniform, which for a rescuer is the same +thing. + +The fix is a choice, and I filed both options as the round-11 ask: +**type-check relation fields at declaration time** (a `sum_lte` over +a declared string should refuse to construct — fail at boot, my +preference) **or wrap evaluation failures** into ValidationError at +call time. Either keeps the promise; the current code keeps neither. +The prober exits 1 until one of them ships, which makes it the +acceptance test, not just the complaint. + +## Notes + +- The oracle is a literal `:allow`/`:reject` column typed by hand. + Deriving expected values from any shared code would let a shared + bug agree with itself — the same discipline as the round-9 torture + test's recomputed concurrency. +- Three probes document semantics rather than test them (missing=0, + presence-not-truthiness, lte-not-lt). Once pinned by a prober, + they stop being implementation accidents and start being contract. + +## Verdict + +The paved road is solid: thirteen probes, zero divergences, and the +contested semantics are now pinned on purpose. Off the road, the new +predicates crash in the wrong uniform. Exit 1 by design — this +prober is the round-11 acceptance test, and it will go green the day +the edge is filed down. diff --git a/docs/perspectives/round-10/07-solnic.md b/docs/perspectives/round-10/07-solnic.md new file mode 100644 index 0000000..3bfedbc --- /dev/null +++ b/docs/perspectives/round-10/07-solnic.md @@ -0,0 +1,68 @@ +# Round 10 field notes — Piotr Solnica diffs the laws + +*Built: `examples/relation_diff.rb` — semver classification for the +rules themselves: tightened limits, widened demands, changed laws, +added and removed rules, and the one honest shrug that remains.* + +## What I built and why + +My round-8 semver advisor ended every report with a qualifier I +hated: "3 breaking changes *in the declarations*." Rules were +lambdas; a diff cannot see inside a lambda; so the most dangerous +class of contract change — policy — was invisible to the one tool +whose job is noticing change. I filed the ask in round 9, the +relations shipped this morning, and this example is the payoff: + +``` +BREAKING rule :fits limit tightened 6000 -> 4000 +BREAKING rule :customs now also demands incoterm +BREAKING rule :one_auth changed LAW: mutually_exclusive -> requires +BREAKING rule :speedy added - a new law callers never agreed to +OPAQUE rule :audited is a lambda in both versions +COMPATIBLE rule :legacy removed +verdict: 4 breaking rule changes -> major version bump +``` + +The classification logic is the same variance reasoning as round 8, +now applied one level up. Rules constrain *inputs*, so they break +when they *tighten*: a lower `sum_lte` limit rejects previously +legal calls; a `requires` that demands one more field fails callers +who satisfied v1; a new rule is a law existing callers never agreed +to. Removal is the loosening direction — every v1-legal call stays +legal — so it's compatible, however alarming a deleted rule looks in +review. + +## The law-change row + +The subtlest classification is `:one_auth`: same rule id, same +fields, but the relation flipped from `mutually_exclusive` to +`requires`. That's not a tightening or a loosening — the two laws +aren't even comparable on one axis ("give at most one" versus "if +one, then both"). The diff refuses to arithmetic it and says what it +is: **a new contract wearing an old name**, breaking by definition. +Tools that force every change onto a tighter/looser spectrum +misclassify exactly these, and these are the ones that page you. + +And the lambda rule still gets the shrug — `OPAQUE, presumed +breaking` — but the meaning of that shrug has inverted. In round 8 +it was a ceiling on the tool; now it's a *choice made per rule*. If +`:audited` mattered to your consumers, you'd declare it as a +relation and it would join the diff. Opacity is now opt-in, which is +the correct default for escape hatches. + +## Notes + +- Presumed-breaking for opaque rules is the only safe default: a + diff that can't see a change must not certify its absence. The + advisor's job is to be conservative exactly where it is blind. +- Fourth derivation tool from the rules metadata in two rounds + (validation, generation, projection, now diffing) — the same + compounding the field declarations showed in rounds 5-8. Predicates + as data pays the same rent schedule. + +## Verdict + +The last opaque corner of the contract now diffs. "Is this breaking?" +covers the declarations *and* the laws over them, with one +honestly-labeled shrug remaining — and even the shrug is a choice +now, not a limitation. diff --git a/docs/perspectives/round-10/08-mperham.md b/docs/perspectives/round-10/08-mperham.md new file mode 100644 index 0000000..2f24336 --- /dev/null +++ b/docs/perspectives/round-10/08-mperham.md @@ -0,0 +1,60 @@ +# Round 10 field notes — Mike Perham gives retries a wallet + +*Built: `examples/retry_budget.rb` — one fleet-wide retry allowance: +transient failures spend from it, hopeless ones can't touch it, and +an empty wallet means failing fast instead of joining the storm.* + +## What I built and why + +A retry storm is the outage you throw yourself, on top of the one +you already have. Every job's retry policy is individually +reasonable — three attempts, backoff, jitter, all the round-5 +hygiene — and collectively insane, because during a real outage +*every* retry is doomed and every one of them costs a timeout, +a connection, and a line item: + +``` +strategy A - every job for itself: 45 calls at a dead host +strategy B - one wallet, 5 retries: 17 calls, 10 jobs failed fast +``` + +Twenty-eight requests deleted, zero value lost — the upstream was +down for all of them. The difference is one idea: **retries are a +shared resource**. Per-job policies answer "should I try again?"; +during an incident the only question that matters is "should +ANYONE?" — and a question about *anyone* needs state that belongs +to *everyone*, which is what the budget is. Round 9's breaker asked +the same question per-upstream; the budget asks it per-window. Both +are fleet-memory where per-job policies have only self-memory. + +## The nil convention, spending department + +The wallet composes with this round's other release: the auth job's +journaled verdict (`retryable: false`) means it never spends from +the budget — not because we're stingy, but because a hopeless +failure retried is a lie told twice, and worse, it *drains the +wallet the transient failures might still need*. Meanwhile a nil +verdict spends normally: suspicion, not a death sentence, exactly +per `TaskFailure#possibly_transient?`. Policy code finally splits +the three-valued verdict at the right joint without every author +re-deriving the joint. + +## Notes + +- The budget class is fifteen lines in the example because it wants + **non-blocking admission**: a `RateLimit` makes you *wait* for + capacity; a budget must tell you *no* right now. Waiting for retry + capacity during an outage would be a queue of doomed requests — + the storm with extra steps. Filed as the round-11 ask: + `RateLimit#try_acquire`, so windowed budgets can be RateLimits and + this class can retire. +- Failing fast when the wallet is empty is not giving up — it's + *believing the fleet's own evidence*. Five doomed retries in one + window is a diagnosis; the eleventh job doesn't need to reconfirm + it at the price of another timeout. + +## Verdict + +45 calls down to 17 with nothing lost, and the deleted 28 were the +ones that would have kept the upstream on its knees. Retry policies +are habits; budgets are decisions. Give the fleet a wallet. diff --git a/docs/perspectives/round-10/09-sandimetz.md b/docs/perspectives/round-10/09-sandimetz.md new file mode 100644 index 0000000..e521ae7 --- /dev/null +++ b/docs/perspectives/round-10/09-sandimetz.md @@ -0,0 +1,72 @@ +# Round 10 field notes — Sandi Metz counts the consumers + +*Built: `examples/rule_shapes.rb` — one policy written three ways +(lambda, structured check, relation), audited by four consumers. +The table is the argument.* + +## What I built and why + +"Express shipments need a customs code" is one sentence of policy, +and this framework now offers three ways to write it down. When a +system gives you three representations of the same thing, that's +not redundancy — it's a design decision it has politely declined to +make *for* you. So the question worth an example is: how do you +choose? + +Not by taste. By **counting who must understand it**: + +``` +shape enforced explains generatable projects +lambda yes no no no +structured check yes yes no no +relation yes yes yes yes +``` + +All three enforce. If enforcement were the whole job they'd be +interchangeable, and style guides would argue about them forever +precisely because nothing real was at stake. But the consumers +differ, and each row is a different answer to "who else gets to +understand this policy?" + +- The **lambda** answers one message — `call` — so it has exactly + one consumer: the validator, at runtime, with real inputs in hand. + Everyone else (the message deriver, the generator, the schema + export, Piotr's diff) gets nothing. Code keeps secrets. +- The **structured check** adds `fields:` and `message:` — metadata + *about* the predicate. Now violations explain themselves and point + at their fields. Two more consumers, same opaque core. +- The **relation** makes the predicate itself data, and the + consumers multiply behind your back: tools that never *run* the + rule can still *read* it. This week alone: Matz asked it as a + question, DHH projected it into a schema, Piotr diffed it across + versions. None of those tools existed when the rule was declared. + That's the tell of a good representation — it keeps answering + questions it wasn't designed for. + +## The principle underneath + +This is the same lesson as my duck-agents parade last round, viewed +from the other side. There, a *narrow message contract* let five +shapes of object walk through one seam. Here, a *rich data contract* +lets one shape of rule serve five kinds of consumer. Both are the +same discipline: decide what must understand what, then choose the +representation that makes those dependencies cheap — messages when +behavior should stay private, data when it must be shared. + +And the closing caveat matters: save lambdas for policies that are +*genuinely* secrets — the fraud heuristic, the pricing curve. An +escape hatch used by default stops being an escape hatch and starts +being a ceiling. + +## Notes + +- The four consumer probes are each five lines and behavioral — the + table's "yes" means a consumer actually extracted value, not that + a capability was advertised. Audits should run, not read. + +## Verdict + +Representation isn't style; it's a decision about who else gets to +understand you. Count the consumers, then choose. Code keeps +secrets, data makes friends — and this framework now lets a policy +pick its social life per rule. diff --git a/docs/perspectives/round-10/10-ankane.md b/docs/perspectives/round-10/10-ankane.md new file mode 100644 index 0000000..e7da072 --- /dev/null +++ b/docs/perspectives/round-10/10-ankane.md @@ -0,0 +1,66 @@ +# Round 10 field notes — Andrew Kane ships the reject file + +*Built: `examples/batch_import.rb` — 500 seeded-dirty rows through +one contract: 382 accepted, 118 rejected with line, field, and rule, +in 81ms.* + +## What I built and why + +Every data tool I've shipped eventually meets the same file: the +customer upload. Typos in enums (`"trian"`), zero weights, columns +that drifted a header to the left, and combinations that are +individually fine and jointly impossible. The two ways importers +die: they **raise on row 37** (an importer that crashes on the +first bad row is a tool for importing 36 rows), or they **write +"invalid row"** in a log (a reject file without reasons is a support +ticket generator). + +The contract turns out to supply both fixes for free: + +``` +accepted: 382 rejected: 118 (500 rows, 81ms, 162us/row) +reject causes: customs 45, mode 36, weight 26, fits 11 +line 12: customs: express requires customs_code +``` + +Collect-don't-crash is just `rescue ValidationError` per row — the +validator reports *every* violation on a row at once (round-5 +behavior), so a row with three problems generates one reject line +with three reasons, not three round-trips through support. And the +reasons are already sentences, because the relations derive their +own messages. + +## The rows only relations catch + +The interesting rejects are `fits: 11` and most of `customs: 45` — +rows where **every column is individually valid**. Weight 4,000: +fine. Volume 4,500: fine. Together: not fine, and no per-column +check — no spreadsheet data-validation dropdown, no CSV linter — +will ever catch it, because the error lives *between* columns. +Cross-field dirt is the dirt that survives all the usual cleaning, +which is exactly why it's the dirt that reaches production. One +declared `sum_lte` caught all eleven. + +Throughput note, because importers are batch jobs: 162 microseconds +a row including the rejection path. Aaron's bench said the same +thing this morning from the other side — at these prices you +validate everything and the bottleneck remains, as always, the +part that talks to the network. + +## Notes + +- The reject file records `line: index + 2` — one-based plus the + header row. Off-by-two line numbers in reject files have burned + more support hours than most bugs; if your reject file says line + 12, pressing ctrl-G 12 in the customer's actual CSV must land on + the bad row. +- The summary histogram (`customs 45 ####...`) is for the engineer; + the per-line file is for support. Same data, two audiences, both + derived — don't make either one read the other's report. + +## Verdict + +An importer is a contract with a patience policy. This one accepts +382 rows, explains 118 rejections down to the rule, and costs less +per row than a DNS lookup. Ship the reject file; your support queue +will send flowers. diff --git a/docs/perspectives/round-8/01-matz.md b/docs/perspectives/round-8/01-matz.md new file mode 100644 index 0000000..bf54d31 --- /dev/null +++ b/docs/perspectives/round-8/01-matz.md @@ -0,0 +1,54 @@ +# Round 8 field notes — Matz plants the forest + +*Built: `examples/plan_forest.rb` — the graph drawn as a forest: roots +at the soil, leaves in the canopy, every task planted at its depth.* + +## What I built and why + +I asked for `stats[:roots]` and `stats[:leaves]` last round; they +arrived, and the metaphor they complete was irresistible. A dependency +graph has always secretly been a garden — things with nothing beneath +them draw from the soil, things with nothing above them face the sun — +and now the framework hands you both lists, so the drawing is a +paragraph: + +``` + (@) preserve jars <- canopy + (@) feast <- canopy + | harvest + | plant rows +\_/ gather seeds <- root +\_/ till the soil <- root +~~~~~~~~~~~~~~~~~~~~ soil +``` + +Depth becomes altitude, `roots` become root systems, `leaves` become +fruit. One glance answers the questions that matter: where does this +plan draw from the world (two roots), what does it produce (two +fruits), how tall did it grow (canopy 5). The gardener's questions ARE +the reviewer's questions; metaphors that survive translation into +arithmetic are the ones worth keeping. + +## Notes on the growing conditions + +- Roots/leaves as *precomputed lists of ids* was the right shape — + fortune teller (round 7) computed them by hand; the forest just + reads them. Two rounds, two tools, and the API converged on what its + users kept deriving. That is how library surfaces should grow: by + paving footpaths, never by paving fields. +- My first draft printed the roots twice — once at their depth, once + at the soil line. The duplication was in my renderer, not the + stats; even simple projections need their draw-each-thing-once + discipline. (Xavier's isomorphism instinct, arriving in my garden.) +- Complete stats census after this round: depth per task, max depth, + max fan-in, roots, leaves. I tried to want more and could not — + everything else I can derive in a line. An API is finished not when + nothing can be added but when additions stop being footpaths. + +## Verdict + +Eight rounds: the plan executes, draws, speaks, tells fortunes, and +now grows. Somewhere along the way this stopped being a test of the +framework and became a small proof about Ruby itself — that a language +optimized for programmer happiness produces libraries you can play +in. The garden was always the point. diff --git a/docs/perspectives/round-8/02-dhh.md b/docs/perspectives/round-8/02-dhh.md new file mode 100644 index 0000000..ca47779 --- /dev/null +++ b/docs/perspectives/round-8/02-dhh.md @@ -0,0 +1,53 @@ +# Round 8 field notes — DHH draws the hill + +*Built: `examples/hill_chart.rb` — Basecamp's hill chart rendered live +from lifecycle hooks: uphill while uncertain, downhill once it's just +execution.* + +## What I built and why + +The hill chart is the only progress visualization I've ever trusted, +because it separates the two things "80% done" conflates: **figuring +out** versus **doing**. Plans have the same split, and the hooks map +onto it without a single judgment call: pending and queued climb the +left slope (waiting on dependencies or a slot — uncertainty you can't +schedule away), `task_slot_acquired` crests the hill, done rolls to +the right base. Three snapshots show the release rolling downhill, +BC→EF→ABCDEF, nobody asked anyone for a percentage. + +The kanban board (round 5) shows *columns*; the hill shows *risk*. +A task stuck uphill for three snapshots is a different conversation +than a task grinding downhill — the first needs unblocking, the +second needs patience. Boards can't say that; hills say only that. + +## The honest-divider argument + +Human hill charts have a known failure mode: people park dots at the +crest because admitting "still uphill" feels like confessing. This +hill can't lie — positions are derived from hook events, and the +crest is *literally* `task_slot_acquired`. When the chart is a +projection of facts rather than a survey of feelings, the pathology +disappears. Same lesson as the check-in (round 7): status extracted +from work beats status reported about work, every time, because it +can't be performed. + +## Notes + +- The surface-following renderer reads the hill's own ASCII art to + find where letters sit — the drawing is data about itself. Silly, + but it meant changing the hill shape is editing a string, not a + coordinate table. +- Mapping choice worth stating: `queued` (scheduled, awaiting a slot + or dependencies) is UPhill. That's the round-4 hook distinction + paying off again — before `task_slot_acquired` existed, queued and + running were indistinguishable, and this chart would have put + blocked work on the downhill side, which is exactly the lie hill + charts exist to prevent. + +## Verdict + +Kanban for columns, hills for risk, check-ins for prose — all three +generated from the same hooks, none requiring a meeting. The project +management suite nobody has to feed is nearly complete; someone +should stop me before I build the Gantt-chart-hater's Gantt chart. +(Aaron already did. It's fine. It's good, even.) diff --git a/docs/perspectives/round-8/03-tenderlove.md b/docs/perspectives/round-8/03-tenderlove.md new file mode 100644 index 0000000..dda7660 --- /dev/null +++ b/docs/perspectives/round-8/03-tenderlove.md @@ -0,0 +1,55 @@ +# Round 8 field notes — Aaron Patterson interrogates the variance + +*Built: `examples/variance_detective.rb` — twenty journaled runs, then +a p90/p50 hunt for the task whose tail betrays it.* + +## What I built and why + +`duration_percentile` shipped this round (my ask from the perf-history +notes), so the detective works the case averages can't crack: + +``` +fetch:profile 22ms 24ms 1.1x +fetch:recommendations 22ms 91ms 4.2x <- SUSPECT +``` + +Same median as its innocent neighbor — **identical p50s** — and a p90 +four times higher. An average would report ~40ms and imply a task +that's uniformly a bit slow; the percentile spread reports the truth: +fine most of the time, terrible 30% of the time, which is the +signature of cold caches, lock contention, or a silently retried +upstream. Flakiness is a *distribution shape*, not a magnitude, and +you cannot see shape in a scalar. + +## Craft notes + +- The p90/p50 *ratio* is the right detector because it's + scale-invariant: a 2ms task with an 8ms tail and a 2s task with an + 8s tail are the same pathology at different magnitudes. Absolute + thresholds would miss one or false-positive the other. +- Twenty runs, not ten. My first draft used ten and the seed + cooperated with the suspect — the 30% slow path fired often enough + to drag p50 itself onto the slow side, and the ratio went quiet. + Percentiles need *samples*; a p90 over ten points is an anecdote + wearing math. This is the flakiness-detection version of the noise + floor lesson: statistical tools have minimum feeding requirements, + and starving them produces confident nonsense. +- The journal made the boring part free: twenty runs accumulated + samples by description with zero harness code, because + `duration_samples` just collects what the fsync was already + writing. The perf suite (Gantt, knee, path, diff, history) now ends + where it should: variance. + +## Where this goes in CI + +Nightly: run the plan N times, journal them, fail if any task's ratio +crosses 3x. Flakiness caught *before* it becomes the intermittent +timeout that eats an on-call week — every infra team has that one +task, and none of them found it from a dashboard of averages. + +## Verdict + +Six performance tools, one journal, and the last one answers the +question that starts the most arguments: "is it slow, or is it +*sometimes* slow?" Those are different bugs with different fixes, and +now they have different numbers. diff --git a/docs/perspectives/round-8/04-fxn.md b/docs/perspectives/round-8/04-fxn.md new file mode 100644 index 0000000..6b10979 --- /dev/null +++ b/docs/perspectives/round-8/04-fxn.md @@ -0,0 +1,61 @@ +# Round 8 field notes — Xavier Noria merges the branches + +*Built: `examples/plan_merge.rb` — a three-way merge of plan wire +formats: independent changes combine, the same seam rewired two ways +is a conflict, reported in topology vocabulary.* + +## What I built and why + +Round 6 made plans serializable, round 7 made them diffable; a format +isn't done until it *merges*, because artifacts that live in version +control get edited on branches. The scenario is the eternal one: ours +adds `dedupe` between parse and rank; theirs adds `moderate` in the +same seam (plus an independent `audit` leaf). The merge: + +``` +cleanly merged: + publish -> audit, + both new stages +CONFLICTS: seam parse -> rank: + ours: parse -> dedupe -> ... + theirs: parse -> moderate -> ... +``` + +The independent change merged silently, as it should. The collision is +reported as what it *is* — two teams rewired the same seam — and the +example says the important sentence out loud: **resolution is a design +decision.** Should content be deduped before moderation or after? No +textual merge algorithm can answer that; the tool's whole job is to +ask the question in vocabulary a human can adjudicate. A line-based +merge of these two JSONs would have either produced a mangled edge +list or, worse, auto-merged both edges in and silently created a graph +where rank has two competing inputs nobody ordered. + +## The confession, promptly + +My first conflict detector found nothing — the replacement-finder +closed over the outer scope's `in_base` (always true at that point) +instead of testing each candidate edge against the base. A shadowing +bug, the classic block-variable kind, and the demo printed "cleanly +merged" over an actual conflict. **A merge tool that under-reports +conflicts is maximally dangerous precisely because its failure mode +looks like success.** The corrected detector was two `base_edges.key?` +calls; the lesson is older: test your tool on the case it exists for +before trusting the case it exists for. + +## Format-trilogy notes + +- Merge operates on the wire format, like the diff — you merge what + you commit. The trilogy (render/invert, diff, merge) is now + complete, and each tool is ~40 lines because the format carries + labels and identity-by-description. +- Rename detection remains the shared blind spot (a renamed task + reads as remove+add in both diff and merge). It's the correct + blind spot to have — heuristic renames in a *merge* tool multiply + the silent-wrongness risk I just demonstrated personally. + +## Verdict + +Plans now have the full version-control lifecycle: serialize, prove, +diff, merge — with conflicts surfaced as design questions. And I've +re-learned, in public, why merge tools are held to the highest +honesty standard of any tooling: their lies arrive wearing green +checkmarks. diff --git a/docs/perspectives/round-8/05-ioquatix.md b/docs/perspectives/round-8/05-ioquatix.md new file mode 100644 index 0000000..5994765 --- /dev/null +++ b/docs/perspectives/round-8/05-ioquatix.md @@ -0,0 +1,61 @@ +# Round 8 field notes — Samuel Williams lets the throttle learn + +*Built: `examples/adaptive_throttle.rb` — an AIMD controller probing +an upstream whose capacity is undisclosed, converging on it from +latency alone.* + +## What I built and why + +Aaron's knee finder (round 4) measures capacity *offline*; production +needs the online version, because upstream capacity isn't a constant — +it's a weather system (noisy neighbors, deploys, regional failover). +The answer is thirty years old: **AIMD**, TCP's congestion algorithm. +Probe up one lane per healthy batch; halve on congestion: + +``` +batch 3 target 3 20ms healthy -> probe up to 4 +batch 4 target 4 50ms congested -> halve to 2 +batch 7 target 4 50ms congested -> halve to 2 +...oscillates around 3.0 - the secret capacity is 3 +``` + +The controller never sees `SECRET_CAPACITY`; it derives it from the +only signal a client legitimately has — its own latency — and the +sawtooth (2→3→4→halve) *is* the discovery, permanently re-verifying +itself. When the upstream degrades to capacity 2 next Tuesday, the +throttle notices within two batches. The static `concurrency_limit` +in your config noticed nothing, ever: it's a guess frozen at deploy +time. + +## Why AIMD and not something cleverer + +Because the sawtooth is a *feature*: the periodic probe upward is how +the controller learns capacity has increased, and the multiplicative +decrease is what makes many independent clients converge to a fair +share without coordinating (the same reason the internet doesn't +collapse). Fancier controllers (gradient, BBR-style) estimate faster +but need better signals; AIMD needs one comparator and one threshold. +Infrastructure defaults should be the dumb thing that provably +converges. + +## Notes for the framework + +- Built entirely in userland: a fresh `Async::Semaphore` per batch is + the "adjustable limiter." That works but re-queues waiters at each + resize; a first-class `limit.resize(n)` on `Agentic::RateLimit` + would make the controller continuous rather than batched. That's my + round-9 ask, and the controller here is its specification. +- The congestion threshold (1.6x base) is doing quiet work — too + tight and healthy jitter reads as congestion (herd of false + halvings), too loose and you camp in the degraded zone. Real + deployments should set it from the journal's p50 history (Aaron's + percentiles), closing the loop between the observability stack and + the control stack. + +## Verdict + +The limiter family now has a missing-manual entry: fixed ceilings for +laws you know, windows for quotas you're billed, and — pattern +demonstrated, primitive requested — adaptation for capacities nobody +will tell you. The network figured this out in 1988; agent frameworks +get to skip the intervening collapse. diff --git a/docs/perspectives/round-8/06-jeremyevans.md b/docs/perspectives/round-8/06-jeremyevans.md new file mode 100644 index 0000000..6ccad8d --- /dev/null +++ b/docs/perspectives/round-8/06-jeremyevans.md @@ -0,0 +1,62 @@ +# Round 8 field notes — Jeremy Evans audits the auditor + +*Built: `examples/journal_audit.rb` — five integrity checks over the +journal itself; a tampered journal with four planted defects yields +seven findings.* + +## What I built and why + +Count the tools that now trust the journal blindly: resume, perf +baselines, variance detection, the weekly check-in, the incident +report. Five consumers, one file, zero verification — a trust +concentration that would fail any security review. So: the audit. +Five checks, each a property the *writer* is supposed to guarantee: + +``` +tampered journal: 7 defect(s) + [well-formed JSON per line] line 6 is not valid JSON + [no success without a start] phantom deploy succeeded without starting + [no double success] task honest work succeeded 2 times + [durations non-negative] time thief has negative duration -3 +``` + +Four acts of tampering, seven findings — the phantom entry tripped +*both* the causality check and the monotonicity check. Overlapping +detectors aren't redundancy to eliminate; they're how real corruption +(which never politely violates exactly one invariant) gets caught by +whichever net it hits first. + +## The checks, and why these five + +1. **Well-formed lines** — the crash-truncation case; the journal's + own design says a torn final line is survivable, so the audit + distinguishes "torn tail" from "garbage in the middle." +2. **Monotonic timestamps** — clock skew or splicing; either way, + downstream ordering assumptions die. +3. **No success without a start** — causality. A phantom success is + exactly what a resume tool would happily skip work for. This is + the check that guards *money*. +4. **No double success** — idempotency-key discipline; a task that + "succeeded twice" means descriptions collided or a writer bug. +5. **Non-negative durations** — feeds Aaron's percentiles; one + negative sample silently poisons every baseline downstream. + +The healthy journal — written by the real machinery — passes clean, +which is the other half of the audit's value: it's a conformance test +for the *writer*, run against actual output rather than fixtures. + +## The meta-point I keep arriving at + +Every round, the pattern is the same: something becomes +infrastructure (the journal), infrastructure accumulates dependents, +and dependents inherit its failures invisibly. The move is always to +write the verifier *before* the failure, while it's cheap and nobody +is panicking. This is the sixth referee tool, and the first one +pointed at our own load-bearing wall. It should run in the incident +report's first line: audit, then replay, then conclude. + +## Verdict + +The journal is now the most-verified file in the project, which is +correct, because it's the most-trusted. Trust and verification should +always arrive in that order and that proportion. diff --git a/docs/perspectives/round-8/07-solnic.md b/docs/perspectives/round-8/07-solnic.md new file mode 100644 index 0000000..0649692 --- /dev/null +++ b/docs/perspectives/round-8/07-solnic.md @@ -0,0 +1,54 @@ +# Round 8 field notes — Piotr Solnica advises the version bump + +*Built: `examples/contract_semver.rb` — two contract versions, every +change classified breaking or compatible from both seats, and the +version bump computed instead of debated.* + +## What I built and why + +Contracts are declarations; declarations can be *diffed semantically*; +and a semantic diff of a contract is a semver verdict waiting to be +computed. Five changes between v1.4.0 and the proposal: + +``` +BREAKING input :weight max tightened 10000 -> 5000 +BREAKING input :customs_code added as REQUIRED +BREAKING output :carrier removed +COMPATIBLE input :mode enum widened (road) +COMPATIBLE output :eta_days added +verdict: 3 breaking -> ship as v2.0.0 +``` + +The rule doing the intellectual work is the **variance asymmetry** +(contravariance, wearing street clothes): inputs break when +*tightened* — previously legal calls get rejected — while outputs +break when *narrowed* — consumers reading `:carrier` now get nil. +Widening an input enum is a gift; widening... removing an output is +a theft. The same category of edit flips polarity depending on which +side of the boundary it touches, which is exactly why humans argue +about "is this breaking?" in every API review: they're arguing from +different seats without naming the seats. The advisor names them. + +## Why this is only possible now + +Every rule in the classifier reads a *declaration*: `required:`, +`enum:`, `min:`/`max:`, output keys. Eight rounds ago these were +comments; today they're data precise enough to compute compatibility +from. This is the fourth tool the declarations have paid for (docs, +schema export, 422s, now semver) — and note that the classifier +needed *zero* new framework support. When your metadata keeps +enabling tools you didn't plan, the metadata's shape is right. + +Blind spot, stated plainly: `rules:` lambdas can't be compared, so a +tightened business rule is invisible to the advisor. Structured rules +narrow the gap (fields and messages diff textually) but the predicate +itself is opaque — same boundary Jeremy's prober works around. The +advisor says "3 breaking changes *in the declarations*"; the honest +reading includes that qualifier. + +## Verdict + +"Is this breaking?" is now a computation with named seats instead of +a meeting with unnamed assumptions. Wire it to CI on contract files +and the changelog writes its own major-version warnings — the +declarations keep paying rent. diff --git a/docs/perspectives/round-8/08-mperham.md b/docs/perspectives/round-8/08-mperham.md new file mode 100644 index 0000000..e71df7c --- /dev/null +++ b/docs/perspectives/round-8/08-mperham.md @@ -0,0 +1,59 @@ +# Round 8 field notes — Mike Perham opens the Dead Letter Office + +*Built: `examples/dead_letter_office.rb` — every failure across three +journaled runs, triaged by most-recent-attempt into requeue, parked, +and recovered.* + +## What I built and why + +Sidekiq's morgue taught me that a dead-letter queue is really three +queues wearing one name, and confusing them is how on-call rotations +die. The office sorts the journal's failures into all three: + +``` +REQUEUE: sync:billing (429 x2), sync:tickets (502) +PARKED: sync:warehouse (401 key revoked - a human must act) +RECOVERED: sync:crm (timed out Monday, fine Tuesday - NOT dead) +``` + +Two decisions carry the design: + +1. **Triage by most recent attempt.** sync:crm failed once and + recovered — paging on it is paging for a ghost. sync:tickets + succeeded once and *then* threw a 502 — its old success excuses + nothing. Both mistakes are common in real DLQs, and both come from + treating failure as a set membership instead of a timeline. The + journal is a timeline; the office just reads it in order and keeps + the last word. +2. **The taxonomy addresses the mail.** Rate limits and 502s go on + the requeue manifest; the revoked key gets parked with "a human + must act." Requeuing an auth failure isn't retrying, it's ritual — + the round-4 lesson (errors testify about their own retryability), + now applied at the *fleet* level across runs instead of inside one + policy. + +## The attempt-count column + +"2 failed attempt(s) on record" on sync:billing is quiet gold: a +letter that's been requeued twice already deserves suspicion the +first-timer doesn't. Real offices should escalate on attempt count — +requeue at 1-2, park-with-review at 3+ — and the journal makes the +count free because every failure was fsynced when it happened. +Escalation policies need memory; the journal *is* memory. + +## Notes + +- One triage decision I made deliberately: the office rebuilds + retryability from the error *type name* in the journal, via an + explicit table. The journal stores `error_type` as a string (it + must — it's JSON), so the mapping lives at read time. An + alternative is journaling `retryable:` at write time from + `failure.retryable?`; that's more honest to the moment of failure + and survives taxonomy renames. Filed as the round-9 ask. + +## Verdict + +Requeue, park, recover — three verbs, correctly assigned, from one +replay. The office closes the failure-handling loop the drills +started: errors testify, policies listen, and now the backlog is +sorted by what the testimony actually said. diff --git a/docs/perspectives/round-8/09-sandimetz.md b/docs/perspectives/round-8/09-sandimetz.md new file mode 100644 index 0000000..b467ec2 --- /dev/null +++ b/docs/perspectives/round-8/09-sandimetz.md @@ -0,0 +1,67 @@ +# Round 8 field notes — Sandi Metz lets the graph write the test plan + +*Built: `examples/graph_to_specs.rb` — an RSpec skeleton generated from +`orchestrator.graph`, where each task's structural role dictates which +examples it owes.* + +## What I built and why + +The hardest question in testing isn't "how do I test this?" — it's +"what deserves a test at all?" People answer it by staring at a blank +spec file and free-associating, which produces suites that test the +easy things thoroughly and the important things by accident. + +But a plan's graph already knows the answer, because *structural role +implies test obligation*: + +- **Roots** own the boundary with the world. They owe a fixture-input + example and a named-error-when-unreachable example, because the + world is the only thing that can surprise them. +- **Joins** (two or more dependencies) owe one context per tributary: + "when sales is missing", "when credits is missing". A join that + fails vaguely — "something was nil" — is a join nobody can debug at + 3am. The labeled edges give each absence case its *name*. +- **Single-dependency tasks** owe exactly one example: the transform. + Their input is another task's output; assert on the shape of + `previous_output` and you're done. +- **Leaves** are promises to the outside. They owe an artifact + assertion, because a leaf nobody reads is a plan nobody needed. + +The generator walks `graph[:order]`, checks each task against +`stats[:roots]`, `stats[:leaves]`, and its dependency count, and +prints the describe blocks. Four tasks became eleven examples, and +not one of them came from free association. + +## The part I care about + +The join rule is the payoff. `needs: {sales: orders, credits: refunds}` +was declared for *wiring* — but the labels turn out to be exactly the +vocabulary the failure cases need. "When credits is missing" is a +sentence a human wrote without knowing they were writing it. That's +what good declarations do: they answer questions you hadn't asked yet. +(Piotr found the same thing this round from the semver seat; the +contract metadata keeps buying tools nobody planned.) + +The generator refuses to write assertions, deliberately. It knows +*what deserves a test*, not *what passes one* — the graph knows +structure, not meaning. A generator that guessed at expectations would +produce green suites that verify nothing, which is worse than no +suite: it's confidence without evidence. + +## Notes + +- `stats[:roots]` and `stats[:leaves]` landed this round (our round-7 + ask) and this is precisely the tool they were asked for. Before, a + consumer had to recompute "empty deps" and "never depended on" — + logic every graph tool would duplicate slightly differently. +- One task can wear two hats — a root that is also a leaf owes both + sets of examples. The generator handles this by accumulation, not + classification: roles are checked independently, never `elsif`'d. + Exclusive categories are how edge cases get orphaned. + +## Verdict + +"What should we test?" is a structural question, and structure is +data now. The graph decided what deserves a test; the human decides +what passes one. That's the right division of labor — the machine +does the enumeration, the person does the judgment. diff --git a/docs/perspectives/round-8/10-ankane.md b/docs/perspectives/round-8/10-ankane.md new file mode 100644 index 0000000..7c25765 --- /dev/null +++ b/docs/perspectives/round-8/10-ankane.md @@ -0,0 +1,68 @@ +# Round 8 field notes — Andrew Kane swaps the scorer, not the harness + +*Built: `examples/eval_scorers.rb` — one eval set scored four ways +(exact, keyword containment, numeric tolerance, judge rubric), with a +scoreboard showing which scorer makes failure mean something.* + +## What I built and why + +Last round I wrote golden-case evals with exact equality and noted +that for LLM-backed capabilities the equality check becomes a scorer — +exact, contains, judge-model — but the harness shape doesn't change. +This round I cashed that claim. The seam is one line: + +```ruby +SCORERS = { + exact: ->(expected, actual) { expected == actual ? 1.0 : 0.0 }, + contains: ->(keywords, actual) { keywords.count { ... }.fdiv(keywords.size) }, + tolerance: ->(spec, actual) { (spec[:value] - actual).abs <= spec[:within] ? 1.0 : 0.0 }, + judge: ->(rubric, actual) { rubric.call(actual) } +} +``` + +Every scorer is `(expected, actual) -> 0.0..1.0`. That's the whole +contract. The judge here is an offline rubric lambda; in production +it's a model call — and *nothing else in the file changes*, which was +the point being tested. + +## The signal-to-noise result + +``` +exact 1/3 pass <- flagged 2 cases; 1 is wording noise +contains 2/3 pass <- flagged only the crash ticket +tolerance 2/3 pass <- same +judge 1/2 pass <- same +``` + +Exact scoring failed the refund ticket because the capability said +"customer reports damaged item, refund requested" instead of my golden +"Damaged item; refund requested". Same meaning, different words — +that's not a regression, that's a scorer measuring the wrong thing. +Meanwhile all three appropriate scorers converged on case 3: crash +tickets score priority 0.3 because the capability has no rule for +crashes. One real failure, unanimously flagged, zero noise. + +This is the practical argument for the scorer seam: it's not about +being *lenient* with fuzzy outputs, it's about making every FAIL in +the report be worth reading. An eval suite people ignore because "the +exact-match ones always fail" is a suite that will miss the crash +ticket too. + +## Notes + +- Scorers return graded scores but the gate is binary (`PASS_AT`). + Keeping those separate matters: the judge can say 0.6 and the + threshold decides. When you later want "quality moved from 0.82 to + 0.74 across the suite," the graded numbers are already there. +- Exit 1 when a real failure exists, same as last round's harness. + Evals that can't fail the build are dashboards, not tests. +- The searchable pattern here is pgvector-era déjà vu: the interface + (`(expected, actual) -> score`) is boring on purpose, so the + ecosystem can supply the interesting parts. + +## Verdict + +The harness shape held: swapping equality for containment, tolerance, +and a rubric touched only the scorer table. Next time quality is in +question, the diff is a scorer entry, not a rewrite — and the round-7 +prediction is now a demonstrated fact instead of a field note. diff --git a/docs/perspectives/round-9/01-matz.md b/docs/perspectives/round-9/01-matz.md new file mode 100644 index 0000000..a3135d9 --- /dev/null +++ b/docs/perspectives/round-9/01-matz.md @@ -0,0 +1,65 @@ +# Round 9 field notes — Matz reads the weather + +*Built: `examples/failure_weather.rb` — three journaled days of +failures, read back as a forecast that knows weather from climate.* + +## What I built and why + +Japanese has a word, 天気雨 — rain falling from a sunny sky. Systems +have it too: the retry that would have succeeded if you'd only waited, +falling right next to the 401 that will never succeed no matter how +long you wait. Operators get soaked by both and can't always tell +which one is on their shoulders. + +The new journal field settles it. Every `task_failed` line now carries +`retryable:`, recorded at the moment the drop fell, from the error's +own testimony. So the report can say something a list of stack traces +never could: + +``` +Monday storm damage (1 structural) digest, backup, invoice +Tuesday storm damage (1 structural) backup, invoice +Wednesday drought continues invoice + +digest: rained earlier this week, clear now - weather does that +invoice: this is not weather, it is climate - 401 key expired +``` + +**Weather passes; climate persists.** A retryable failure is weather — +bring an umbrella (a retry policy) and go about your day. A +non-retryable failure is climate — no forecast fixes a drought; +someone must dig a well (rotate the key). The six rainy events split +exactly: three weather, three reports of the same drought. + +## What pleased me + +The metaphor required no force. I did not bend the framework to fit +the weather report; the framework's own distinction — `retryable?`, +asked of every error, journaled when fresh — *is* the +weather/climate distinction. When a metaphor maps without residue, +that usually means the underlying concept was carved at a real joint. + +And the kindness of it: "digest: rained earlier this week, clear now" +is a sentence that lowers a reader's heart rate. The same information +as `LlmRateLimitError (resolved)`, but it treats the operator as a +person who was worried, not a parser of taxonomies. Error messages +are the user interface of failure; they should be written by hosts, +not by stack traces. + +## Notes + +- I triaged Wednesday's sky by mixing counts: `climate > 0 && weather > 0` + is storm damage, climate alone is drought, weather alone is showers. + Four skies covered every day I could script. Small vocabularies + that cover the space completely are a joy. +- The one wrinkle: a `nil` verdict (an error with no opinion) is + neither weather nor climate. My report would silently drop it — + fine for a demo, rude in production. Fog, perhaps. A forecast + should admit when it does not know. + +## Verdict + +The journal learned to say *why it rained*, so the report can say +*whether to wait or to dig*. That is the whole art of operating +systems under failure, expressed as a weather segment. I smiled +writing it, which remains my favorite metric. diff --git a/docs/perspectives/round-9/02-dhh.md b/docs/perspectives/round-9/02-dhh.md new file mode 100644 index 0000000..1f64849 --- /dev/null +++ b/docs/perspectives/round-9/02-dhh.md @@ -0,0 +1,65 @@ +# Round 9 field notes — DHH turns the traffic dial + +*Built: `examples/traffic_dial.rb` — a canary rollout as a single +resized `RateLimit`: dial up on health, dial back on a burned SLO, +hold and page a human after two burns.* + +## What I built and why + +Everyone's rollout story has become a platform: a feature-flag +service, a progressive-delivery controller, a Slack app that asks +permission to proceed. Meanwhile the actual idea is one sentence — +*give the new code a little traffic, and more only if it behaves.* + +That sentence is a `RateLimit` with a `resize` method: + +``` +1 1 lane 20.1ms healthy - dial up to 3 +2 3 lanes 20.1ms healthy - dial up to 6 +3 6 lanes 80.1ms SLO burned - dial BACK to 3 +5 6 lanes 80.1ms SLO burned - dial BACK to 3 +6 3 lanes 20.2ms holding at 3 - stage 6 burned twice; + page the author, not the dial +``` + +v2 hides the classic regression: fine at staging concurrency, +quadruple the latency above 3 in flight. The dial found it at stage +3, backed off, gave it one more honest chance, and then *stopped +experimenting*. That last part matters as much as the backoff. A +controller that keeps re-running a failed experiment isn't +persistent, it's a metronome. Two burns means the code is wrong, and +no amount of dialing fixes wrong code — that's a human's job now. + +## Conceptual compression + +Count the concepts: one limiter, one SLO number, one stage ladder, +one burn counter. That's the entire deployment strategy, and every +piece is visible in forty lines you own. The platform version has the +same four concepts — it just distributes them across three vendors +and a YAML dialect, and then charges you to see the burn counter. + +`resize` is what made this a knob instead of an architecture. Before +this round you'd rebuild the limiter each stage and re-share it with +every client — plumbing pretending to be a feature. Now the clients +hold one object for the lifetime of the rollout and the *controller +moves the ceiling under them*, mid-flight. That's the correct +division: clients know nothing about rollouts; the rollout knows +nothing about clients. + +## Notes + +- The SLO is p50 against a budget, per stage. I resisted p99 — + twelve requests per stage is a demo; pretending to a p99 with + n=12 is how dashboards lie. Use the statistic your sample size + can carry. +- Stage ladder `[1, 3, 6, 10]`, not linear. Rollouts should spend + their caution early, where the blast radius is small and the + information per request is highest. + +## Verdict + +Gradual rollout is a solved problem that keeps getting unsolved by +tooling. One resized limiter did the whole job: found the hidden +regression, contained it to 6 lanes for two brief windows, and ended +the day with a specific bug report instead of an outage. Ship the +fix, turn the dial again tomorrow. diff --git a/docs/perspectives/round-9/03-tenderlove.md b/docs/perspectives/round-9/03-tenderlove.md new file mode 100644 index 0000000..8916200 --- /dev/null +++ b/docs/perspectives/round-9/03-tenderlove.md @@ -0,0 +1,63 @@ +# Round 9 field notes — Aaron Patterson finds the knee + +*Built: `examples/throughput_knee.rb` — one limiter swept from ceiling +1 to 8 via `resize`, two clocks per job, and the exact ceiling where +more concurrency stops buying throughput.* + +## What I built and why + +Every team eventually has the "just raise the concurrency" meeting. +Someone raises it, latency gets worse, someone else lowers it too far, +and the final number is whoever argued loudest. I wanted the number to +come from a sweep instead: resize one limiter through every ceiling, +push the same 24 jobs through each, measure, done. + +``` +ceiling jobs/sec service p50 total p50 +4 198.5 20.1ms 80.3ms <- the knee +5 132.8 40.1ms 100.2ms +8 79.8 100.1ms 140.2ms +``` + +The upstream secretly runs 4 in parallel. The sweep found it without +being told — and `resize` is what made the sweep honest: one limiter +object, eight ceilings, no rebuild-and-reshare between rows. + +## Two clocks or you're lying to yourself + +The design decision that matters: every job gets timed twice. +*Service time* starts when the limiter admits you; *total time* +starts when you submit. The difference is your queue — the wait you +can see. Below the knee, service time is flat at 20ms and total time +shrinks as lanes open: your queue is draining faster. Above the knee, +**service time itself rises** — the queue didn't disappear, it moved +to the server's side of the wire, where your metrics can't see it +and your timeouts will misattribute it. + +That's the diagnostic worth memorizing: *when raising your ceiling +raises the server's latency, you found their ceiling, not yours.* + +And one measurement corrected my own script's prose: I wrote +"throughput goes flat past the knee" and the table said it *falls* — +199 → 133 → 80 jobs/sec. Of course it falls: this upstream degrades +everyone under overload, not just the excess. Flat is the best case; +falling is the common one. The chart knew better than I did, which +is the whole reason to make the chart. + +## Notes + +- The knee detector is one line: the first ceiling whose successor + gains less than 8%. Crude, but it only has to beat "whoever argued + loudest," and it does. +- Samuel's adaptive throttle (round 8) and this sweep are the same + physics, opposite instruments: his AIMD *tracks* the knee + continuously in production; mine *maps* it once on a bench. You + want the map to set the initial ceiling and the tracker to follow + the knee when it moves. + +## Verdict + +Ceiling arguments are now a bench run: sweep, watch both clocks, +read the knee off the table. The polite move past someone else's +ceiling is to stop pushing — and now you know exactly where their +ceiling is. diff --git a/docs/perspectives/round-9/04-fxn.md b/docs/perspectives/round-9/04-fxn.md new file mode 100644 index 0000000..9ef4fa5 --- /dev/null +++ b/docs/perspectives/round-9/04-fxn.md @@ -0,0 +1,66 @@ +# Round 9 field notes — Xavier Noria proves the promises + +*Built: `examples/graph_invariants.rb` — seven invariants of the +`graph` reflection API, proved across four plan shapes including a +deliberate cycle. Exit 0 is a certificate.* + +## What I built and why + +Eight rounds of tools now lean on `orchestrator.graph`: the forest +drawing, the spec generator, the structural diff, the three-way +merge. Every one of them assumes things the documentation *asserts* — +order respects edges, roots have empty dependencies, depth is the +longest path from a root, labels ride their edges. Assertions in +YARD comments are wishes. I wanted proofs: + +``` +chain / diamond / forest / cycle: + order is a permutation of the task set proved + order respects every edge (acyclic only) proved + roots are exactly the tasks with no deps proved + leaves are exactly the tasks nothing feeds proved + depth is 1 + max dependency depth (acyclic) proved + max_depth / max_fan_in agree with sources proved + every needs: label appears on its edge proved +26 proofs, 0 violations +``` + +Each invariant is a lambda from graph to violations; each plan shape +is chosen to stress a different clause — the diamond for labeled +joins, the forest for multiple roots and an orphan, the cycle for +the degenerate case every reflection API secretly dreads. + +## The prover's first catch was its author + +My initial depth invariant ran on all four shapes, and the cycle +violated it: `depth[x] = 1, expected 3`. I stared at the "bug" for a +minute before seeing it was in *my invariant*, not the framework: +depth means "longest path from a root," and a cyclic graph has no +such number — the definition chases its own tail (x's depth needs +y's, y's needs x's). The framework's Kahn's-order computation gives +cycle members *some* finite value; any value satisfies no meaningful +contract. + +The correct fix was not code but *scope*: the invariant's name now +carries "(acyclic only)", the same qualifier the edge-ordering +invariant already had. This is a lesson I keep re-learning from +Zeitwerk: the hardest part of a contract is not enforcing it but +stating its domain precisely. An unscoped promise is a bug that +hasn't picked its reporter yet. + +## Notes + +- The needs-label invariant cross-checks two representations of the + same fact (`needs:` hash vs edge labels). Redundant representations + are where reflection APIs rot first, because nothing forces them to + agree — except now something does. +- Everything here belongs in the spec suite eventually. An example + makes the argument legible; a spec makes it permanent. Filed as a + thought, not an ask — the round-9 spec file already covers the new + surface, and porting these seven lambdas is mechanical. + +## Verdict + +The reflection API's promises are now theorems with a runnable proof, +and the proof's first victim was my own imprecision about cycles — +which is exactly what provers are for. Exit 0, certificate issued. diff --git a/docs/perspectives/round-9/05-ioquatix.md b/docs/perspectives/round-9/05-ioquatix.md new file mode 100644 index 0000000..b395371 --- /dev/null +++ b/docs/perspectives/round-9/05-ioquatix.md @@ -0,0 +1,64 @@ +# Round 9 field notes — Samuel Williams shares the door fairly + +*Built: `examples/fair_share.rb` — two tenants behind one global +ceiling: request-fairness vs tenant-fairness, and live share +rebalancing via `resize`.* + +## What I built and why + +The most common multi-tenancy bug isn't a race — it's a definition. +A FIFO semaphore is perfectly fair *to requests*. But tenant A hires +six workers and tenant B hires two, so "fair to requests" resolves, +silently, to "A gets triple." Nothing errors. Nothing pages. B just +runs at half its entitlement forever: + +``` +no shares, one door for all A: 76 B: 24 <- B wants 48 +2/2 shares, same greedy A A: 52 B: 48 <- B whole again +B idle, static 2/2 shares A: 52 B: 0 <- 2 lanes stranded +B idle, shares rebalanced 4/1 A: 98 B: 0 <- lent, live +B returns, back to 2/2 A: 52 B: 48 <- returned, live +``` + +The fix is the composition we built in round 7: each tenant acquires +its *own share first*, then the shared door — `share_a.and(global)`. +Phase 2 shows what that buys: B reaches its full 48 no matter how +many workers A hires, and A soaks up whatever remains. Fairness +between tenants, work-conservation within the door. + +## What resize adds + +Static shares have a tax, and phase 3 states it plainly: B goes idle +and B's two lanes serve *nobody* — the global door runs half empty +while A queues. Before this round, the options were ugly: rebuild +the limiter objects (and re-hand them to every client holding one) +or over-provision the global and pray. + +`resize` makes shares a *dial on a live object*. Phase 4 lends B's +spare lane to A mid-flight — A jumps to 98 — and phase 5 takes it +back the moment B returns. The composition never changes shape; no +fiber holding a limiter notices anything except capacity arriving or +draining away. That's the property I care most about: the *topology* +of the limiter graph is fixed at boot, and only the *numbers* move +at runtime. Topology changes race; number changes don't. + +## Notes + +- My first draft gave B a single worker, and the numbers refused to + cooperate: B served 24 in phase 1 *and* phase 2 — no starvation, + because one serial worker can only ever want one lane. The chart + was right and my story was wrong: starvation requires unmet demand, + so B needed two workers wanting two lanes. This is now the fourth + round in which a tool corrected its author before the user saw it. +- Order matters in the composition: own share first, then the door. + Acquire them the other way and an over-subscribed tenant holds + global capacity while waiting on its own share — the deadlock-shaped + version of the same idea. + +## Verdict + +Request-fairness and tenant-fairness look identical until the tenants +are unequal, which is always. Composition makes the distinction +enforceable, and resize makes it affordable — the idle tenant's lanes +no longer cost the busy one anything. Fairness as an adjustable +object, not a config value baked at boot. diff --git a/docs/perspectives/round-9/06-jeremyevans.md b/docs/perspectives/round-9/06-jeremyevans.md new file mode 100644 index 0000000..8f00998 --- /dev/null +++ b/docs/perspectives/round-9/06-jeremyevans.md @@ -0,0 +1,54 @@ +# Round 9 field notes — Jeremy Evans tortures the new knob + +*Built: `examples/resize_torture.rb` — three assaults on +`RateLimit#resize`: jagged ceiling epochs under saturating load, a +mid-flight shrink, and a grow that must wake its waiters. Exit 0 is +a certificate.* + +## What I built and why + +A method that mutates a synchronization primitive while fibers are +blocked on it is the most dangerous kind of convenience: trivially +easy to call, and its failure modes only appear under contention, +which is exactly where nobody is looking. Before I use `resize` in +anything I care about, I want its guarantees stated and then attacked. + +The three guarantees, as I reconstructed them from the docs, and the +attack on each: + +1. **An epoch's ceiling binds every admission inside it.** Resize + through jagged ceilings (1→5→2→4→1→3), saturate each epoch with 4× + more jobs than lanes, and recompute max concurrency myself — I do + not trust `high_water` for this, since it's maintained by the same + code under test. All six epochs held exactly at their ceiling. +2. **Shrink drains; it does not evict, and it does not leak.** Fill + five lanes, shrink to 2 while all five run, then submit a second + wave. Every wave-2 admission observed ≤ 2 concurrent: the old + holders finished undisturbed and nothing new joined them above the + new mark. +3. **Grow wakes the queue.** One lane, three long jobs. On the serial + schedule the third admission lands at ~100ms. Grow to 3 at 10ms + and the last admission lands at 10ms — the waiters were resumed by + the resize, not left dozing on the old schedule. This one + exercises `Async::Semaphore#limit=`'s wake-up path specifically, + which is the part I'd least like to discover broken in production. + +## Notes + +- My first run crashed — assault 3 read its start-time variable + before assigning it, because I spawned the jobs and *then* set the + clock. A torture test's first victim is reliably its own harness; + the discipline is fixing the harness without softening the assault. +- What I deliberately did not certify: fairness of wake-up order on + grow (FIFO vs arbitrary), and windowed-mode resize under a sleeping + waiter (the new ceiling applies on next admission check, which can + be up to a full window late). Both are documented behavior, not + bugs — but a certificate should say what it doesn't cover, so this + one does. + +## Verdict + +Three assaults, zero cracks. Shrink drains, grow wakes, ceilings +bind. `resize` earns its place — and the sentence I'll reuse +elsewhere: a mutation method on a concurrency primitive without a +torture test is a data race with a friendly name. diff --git a/docs/perspectives/round-9/07-solnic.md b/docs/perspectives/round-9/07-solnic.md new file mode 100644 index 0000000..1af9785 --- /dev/null +++ b/docs/perspectives/round-9/07-solnic.md @@ -0,0 +1,63 @@ +# Round 9 field notes — Piotr Solnica derives the examples + +*Built: `examples/contract_fixtures.rb` — fixtures generated from +contract declarations (minimal and maximal per capability), proved +against their own validator, with a mutant to prove the validator +still has teeth.* + +## What I built and why + +Every README has an example payload, and every example payload is +lying by the second release. Not maliciously — entropically. The +contract moved and the prose didn't, because nothing *runs* the +prose. + +The fix follows from a principle this framework has been converging +on for six rounds: **anything declared can be derived from.** A +declaration rich enough to *reject* a payload is rich enough to +*construct* one: + +- `enum:` → its first member (`"air"`) +- `min:`/`max:` → the midpoint (`2500` — legal by construction, and + visibly *inside* the range rather than tremblingly on its edge) +- `required:` → membership in the minimal fixture +- everything else → a string that names its own key + +Two fixtures per capability: **minimal** (required keys only — the +smallest legal call, which is what a new integrator actually wants +to see) and **maximal** (every declared key — what a code generator +wants). Then the part that makes it an engineering artifact instead +of a convenience: the referee validates every generated fixture +against the same contract it came from, and then drops a required +key to prove the validator still rejects things. A generator proved +only against an accept-everything validator has proved nothing — +the mutant is what makes the green checkmarks mean something. + +Handwritten examples are promises; derived ones are consequences. + +## The blind spot, on the record + +`rules:` are predicates, and a generator cannot see inside a lambda. +A fixture that is legal per-field can still violate a cross-field +rule — my midpoint weight and first-enum mode could, in some +contract, be exactly the forbidden combination. The example prints +this caveat in its own output because a derivation tool that +overstates its coverage is worse than none. + +This is now the third tool to hit the same wall (Jeremy's prober +works around it dynamically, my semver advisor stated it, now the +generator inherits it), and the shape of the fix is visible: +structured rules already carry `fields:` and `message:`; if the +common cases also carried a machine-readable *relation* (`sum_lte:`, +`requires:`, `mutually_exclusive:`), the generator could satisfy +them and the advisor could diff them. Filing that as the round-10 +ask: **relation-typed structured rules** — keep the lambda escape +hatch, but let the declarable majority be declared. + +## Verdict + +The contract now produces its own documentation examples, and proves +them on every run. Third derivation tool this quarter from the same +metadata (docs, schemas, semver, now fixtures) — when declarations +keep compounding like this, the metadata design has paid for itself +several times over. diff --git a/docs/perspectives/round-9/08-mperham.md b/docs/perspectives/round-9/08-mperham.md new file mode 100644 index 0000000..c4f0951 --- /dev/null +++ b/docs/perspectives/round-9/08-mperham.md @@ -0,0 +1,64 @@ +# Round 9 field notes — Mike Perham installs the breaker + +*Built: `examples/circuit_breaker.rb` — a closed/open/half-open +breaker in two acts: a 503 outage handled with three strikes, and a +revoked key handled with one.* + +## What I built and why + +Retries handle failures one call at a time. But an outage isn't one +call's problem — it's a *condition*, and treating a condition with +per-call optimism means every request pays the full timeout to +rediscover what the last request just learned. The breaker is the +missing memory: after enough strikes, stop asking. + +``` +act one: 503s, ticks 4-9 act two: 401, tick 2 onward + 4 failed (retryable: true) 2 failed (retryable: false) + 5 failed (retryable: true) - breaker TRIPS + 6 failed - breaker TRIPS 3 SKIPPED + 7 SKIPPED 4 SKIPPED + ... ... + 10 probe SUCCEEDED - closes 6 probe fails - TRIPS again +``` + +Act one is the textbook: three strikes, trip, eat the middle of the +outage as skips (nothing sent, nothing billed, no timeout waited +out), one probe to discover the recovery. Six ticks of outage, three +calls felt it. The gap is the money — and at LLM prices, the gap is +real money. + +## The verdict feeds the trip decision + +Act two is why this example belongs in round 9. Strike counts exist +because a 503 *might* pass — you give it three chances to be +transient. But the 401's journaled verdict says `retryable: false`: +the error itself testified that no retry can ever help. Giving that +error three strikes isn't caution, it's ritual. The breaker trips on +first contact, and each half-open probe re-trips on the same wall +until a human rotates the key. + +The breaker reads the verdict from the journal — recorded at write +time, this round's release — rather than re-deriving it from a +class-name table. Same argument as the dead letter office: the +moment of failure is when retryability is known most precisely; +everything derived later is reconstruction. + +## Notes + +- My first draft read `events.last[:retryable]` and got nil — the + last event after a failed run is `plan_completed`, which has no + verdict. The nil then hit the `retryable ? 1 : TRIP_AFTER` branch + and instantly tripped on a *transient* 503. Two lessons in one bug: + scan for the event you mean, and never let "no opinion" silently + mean "hopeless." A production breaker should treat a nil verdict + as retryable-with-suspicion, not as a death sentence. +- Half-open admits exactly one probe. Letting the whole queue through + on recovery is how you re-kill the thing that just got back up. + +## Verdict + +Retry policies answer "should THIS call try again?"; breakers answer +"should ANYONE?" With retryability journaled at the moment of +failure, both answers now come from the error's own testimony — +counted strikes for the maybes, an instant trip for the nevers. diff --git a/docs/perspectives/round-9/09-sandimetz.md b/docs/perspectives/round-9/09-sandimetz.md new file mode 100644 index 0000000..201117b --- /dev/null +++ b/docs/perspectives/round-9/09-sandimetz.md @@ -0,0 +1,68 @@ +# Round 9 field notes — Sandi Metz counts the ducks + +*Built: `examples/duck_agents.rb` — one plan, five differently-shaped +agents through the single `agent:` seam: lambda, instance, Method +object, curried proc, and a decorator that wraps any of them.* + +## What I built and why + +Eight rounds of examples have passed lambdas to `agent:` so uniformly +that a reader could be forgiven for thinking the seam *requires* +lambdas. It doesn't, and the difference between "we always pass +lambdas" and "you must pass lambdas" is the difference between a +convention and a cage. I built the parade to prove the cage isn't +there: + +``` +fetch lambda -> {records: 12, ...} +dedupe instance with #call -> {unique: 9, pass: 1} +stats Method object -> {mean: 4.2, max: 9} +render curried proc -> {rendered: true, format: "html"} +audit decorator around a lambda -> {audited: true, timed_ms: 0.0} +``` + +The seam asks one question — *can you be called with a task?* — and +never asks anyone's class. So every shape that answers walks in, and +each shape earns its place for a different reason: + +- The **lambda** for logic too small to deserve a name. +- The **instance** when logic deserves a home — my `Deduper` carries + state (`pass: 1`) that a lambda would have to smuggle in a closure. +- The **Method object** when the logic already lives somewhere — + `Stats.method(:summarize)` joins the plan without a wrapper, which + means no wrapper to test. +- The **curried proc** for configuration applied ahead of time: + `render.curry["html"]` is dependency injection wearing a very small + hat. +- The **decorator** is the payoff of all of it: `Timed` consumes the + same one-message contract it provides, so it stacks around *any* of + the other four without knowing or caring which it got. + +## The design lesson + +Depend on messages, not classes, and your plugin API is every object +ever written — including the ones not written yet. The framework got +this right in a way worth naming precisely: `resolve_agent` checks +`respond_to?(:execute)` and otherwise wraps the callable. Two ducks +accepted, both by *capability*, neither by ancestry. There is no +`AgentBase` to inherit, and so there is no inheritance debt to +pre-borrow — the round-1 sin (`FactoryMethods` breaking under +subclassing) has stayed fixed at the seam that matters most. + +The decorator deserves one more sentence. That `Timed` works on all +five shapes isn't a feature anyone built; it's a *consequence* of the +narrow contract. Wide interfaces make decorators expensive (forward +everything!); one-message interfaces make them nine lines. If you +want composition, keep your contracts poor. + +## Notes + +- I resisted adding a sixth duck that type-checks its input, as a + cautionary contrast. The parade argues better without the clown. + +## Verdict + +The seam is honest: it asks for what it needs — one message — and +not for who you are. Five shapes, zero adapters, one nine-line +decorator that fits them all. That's what "design for change" looks +like when it's cheap. diff --git a/docs/perspectives/round-9/10-ankane.md b/docs/perspectives/round-9/10-ankane.md new file mode 100644 index 0000000..53edf74 --- /dev/null +++ b/docs/perspectives/round-9/10-ankane.md @@ -0,0 +1,62 @@ +# Round 9 field notes — Andrew Kane runs the shootout + +*Built: `examples/impl_shootout.rb` — two candidate implementations +of one capability, one eval set, and a two-axis verdict: accuracy +AND latency, on the same table.* + +## What I built and why + +Every capability eventually has a challenger: the regex that shipped +in an afternoon versus the subtler thing someone wrote on a weekend. +The upgrade decision then happens in a meeting, powered by whoever +has the freshest anecdote. I wanted the decision to be a table: + +``` +v1 regex accuracy 63% p50 2.1ms +v2 weights accuracy 100% p50 10.1ms +verdict: v2 wins 8/8 to 5/8 - and costs 5x the latency +``` + +The eval set is the referee, and the deciding cases share one shape: +"password reset email shows an error page" has one bug word and five +points of account evidence. v1 answers by *first match* — clause +order, an accident of code layout — and files it under bug. v2 +answers by *total evidence* and files it under account. That +difference is invisible in a demo and everywhere in production, +because production tickets are all like this: multi-topic, casually +worded, keyword-colliding. + +## Both axes or it's marketing + +The scoreboard refuses to report accuracy without latency. "v2 is +better" is not a finding; "v2 buys 3 more correct routes per 8 +tickets at 8ms each" is, because it's *deniable* — a team routing a +million tickets a day at tight budgets can look at the same table +and correctly choose v1. Benchmarks in my gems' READMEs follow the +same rule: publish the numbers that let someone reasonably choose +the competitor, or you've published an ad. + +Two implementation notes that carried weight: + +- v2's first draft matched exact words and *lost* the crash case — + "crashes" isn't "crash." Prefix-stem matching fixed it, which is + the eval set doing its actual job: catching the bug in the + challenger before the meeting where you propose it. +- A perfect 8/8 doesn't mean v2 is done; it means the eval set + **stopped discriminating**. The scoreboard says so in its own + output. Grow the set until your best candidate fails — a suite + your champion aces is a suite that's stopped asking questions. + +## Notes + +- The shootout composes with round 8's scorer seam: swap `correct:` + for a graded scorer and this same harness A/Bs LLM-backed + candidates, where "accuracy" becomes mean score and latency + becomes dollars. Nothing about the table changes shape. + +## Verdict + +Upgrade debates become table reads: two candidates, one eval set, +both axes visible. v2 wins this one on evidence-versus-clause-order, +and the latency price is printed next to the win so the choice stays +a choice. diff --git a/examples/README.md b/examples/README.md index b4c16df..a4be485 100644 --- a/examples/README.md +++ b/examples/README.md @@ -6,55 +6,82 @@ not this file. | Example | What it shows | |---------|---------------| +| `adaptive_throttle.rb` | The Adaptive Throttle: nobody TELLS you an upstream's capacity - you discover it. An AIMD controller (TCP's algorithm) p... | | `api_reference.rb` | The API Reference Generator: walk the registry, emit reference docs for every capability - types, enums, bounds, policie... | | `backoff_conformance.rb` | Backoff Conformance: every strategy x jitter combination, a thousand draws each through an injected seeded RNG, checked ... | +| `batch_import.rb` | The Batch Import: 500 rows of the kind of data people actually upload - typos, header drift, impossible combinations - r... | | `burst_absorber.rb` | The Burst Absorber: three waves of requests slam a credential with a ceiling of 3 (Agentic::RateLimit - this round's rel... | +| `cancel_drill.rb` | The Cancel Drill: structured concurrency's core promise is that cancellation is PROMPT - stop means stop, not "finish ev... | | `capability_evals.rb` | Capability Evals: golden test cases run against registered capabilities, scored, and gated. When you swap a lambda for a... | | `changelog_scout.rb` | The Changelog Scout: reads real git history, classifies every commit through a contract-checked capability, and drafts t... | +| `circuit_breaker.rb` | The Circuit Breaker: when an upstream is down, the cheapest request is the one you don't send. The breaker trips after 3... | | `collaboration_tracer.rb` | The Collaboration Tracer: lifecycle hooks record every message the orchestrator sends and every reply that comes back, t... | | `command_bus.rb` | The Command Bus: every command is a composed capability with its OWN declared contract (new in this round - compositions... | | `composed_limits.rb` | Composed Limits: a real provider enforces BOTH a billed quota and a connection ceiling. quota.and(pool) - new this round... | +| `contract_fixtures.rb` | Contract Fixtures: example payloads in docs rot the day the contract changes. So don't write them - DERIVE them. This ge... | | `contract_fuzzer.rb` | The Contract Fuzzer: for every registered capability, generate inputs that SHOULD pass its declared contract and mutatio... | +| `contract_overhead.rb` | The Contract Overhead Bench: "should we validate every call?" is a performance question, so answer it with a measurement... | +| `contract_semver.rb` | The Contract Semver Advisor: two versions of a capability's contract, every change classified as breaking or compatible ... | | `cost_estimator.rb` | The Cost Estimator: price the plan BEFORE running it - per-task token estimates times a pricing table - gate on budget, ... | | `coupling_cartographer.rb` | The Coupling Cartographer: which files lean on which? Every file is surveyed for the constants it DEFINES and the consta... | | `critical_path.rb` | The Critical Path: after a run, combine the graph topology with measured durations to find the chain of tasks that deter... | +| `dead_letter_office.rb` | The Dead Letter Office: three days of journaled runs, every failure collected and triaged by what the errors said about ... | | `deploy_train.rb` | The Deploy Train: lint -> test -> build -> canary -> ship, where a red gate stops the train and everything behind it rep... | | `doc_coverage.rb` | The Documentation Surveyor: measures YARD comment coverage for every public method in a lib/ tree. One survey task per f... | +| `duck_agents.rb` | Duck Agents: the agent: seam asks one question - "can you be called with a task?" - and five differently-shaped objects ... | | `dungeon_crawl.rb` | The Dungeon Crawl: a quest is a plan, rooms are tasks, and doors are dependencies. The map is drawn from the orchestrato... | | `durable_batch.rb` | The Durable Batch: six billable "LLM calls" run under an ExecutionJournal. Mid-batch, the process dies for real - exit!,... | | `error_taxonomy_drill.rb` | The Error Taxonomy Drill: three tasks fail three different ways - a rate limit (retryable, says the error itself), an au... | +| `eval_scorers.rb` | Eval Scorers: the same eval set scored four ways - exact match, keyword containment, numeric tolerance, and a judge rubr... | | `exquisite_corpse.rb` | The Exquisite Corpse: three artists each draw one part of a creature without seeing the others' work; the assembler rece... | +| `failure_weather.rb` | The Failure Weather Report: a journal of three days, read as a forecast. Retryable failures are WEATHER - showers that p... | +| `fair_share.rb` | Fair Share: two tenants, one upstream. The global ceiling is fair to REQUESTS - first come, first served - but tenant A ... | | `flaky_api_drill.rb` | The Flaky API Drill: a task that times out twice before succeeding, run under a retry policy with exponential backoff an... | | `form_errors.rb` | The 422 Generator: turn a ValidationError into the API error document your frontend actually wants - message, allowed va... | | `freight_rules.rb` | The Freight Desk: a quoting capability whose tariff book is written as cross-field contract rules (new this round). Per-... | | `gem_scout.rb` | Gem Scout: describe what you need, get a ranked shortlist of gems. Search and scoring are separate capabilities; the sea... | | `graph_critic.rb` | The Graph Critic: reviews a plan's dependency structure BEFORE it runs, the way you'd review a class diagram. God tasks,... | +| `graph_invariants.rb` | The Graph Invariants Prover: the reflection API makes promises - order respects edges, roots have no dependencies, depth... | | `graph_style.rb` | The Graph Style Guide: RuboCop for plans. Cops with thresholds run against any orchestrator's graph - depth, fan-in, orp... | +| `graph_to_specs.rb` | Graph to Specs: the plan's structure dictates its test plan - roots need fixture cases, joins need one case per missing ... | | `haiku_agent.rb` | The three-line agent. Run me with no API key at all: | +| `hill_chart.rb` | The Hill Chart: Basecamp's answer to "how's it going?" - work climbs the hill while it's still uncertain (queued, waitin... | +| `impl_shootout.rb` | The Implementation Shootout: two candidates for the same capability, one eval set, and a verdict computed instead of vib... | | `incident_report.rb` | The Incident Report: a nightly batch dies at 3am. The on-call's first three questions - what ran? what broke? what do I ... | | `invariant_sentinel.rb` | The Invariant Sentinel: domain invariants checked after EVERY task, from a lifecycle hook. When a task leaves the world ... | | `jitter_shootout.rb` | The Jitter Shootout: none vs equal (+/-25%, the default) vs full (uniform over [0, delay], new this round) - same forty ... | +| `journal_audit.rb` | The Journal Audit: seven tools now trust the journal, so the journal itself gets audited - well-formed lines, monotonic ... | | `json_schema_export.rb` | The Schema Export: a capability contract emitted as draft-07 JSON Schema (new this round), then PROVEN faithful - the sa... | | `kanban_board.rb` | The Kanban Board: a plan rendered as the three columns everyone actually understands - To Do, Doing, Done - reprinted at... | | `knee_finder.rb` | The Knee Finder: runs the same plan at increasing concurrency limits, measures wall time and total queue-wait via the ta... | | `latency_lab.rb` | The Latency Lab: 20 simulated LLM calls (200ms of IO each) executed through the orchestrator at different concurrency li... | | `live_dashboard.rb` | The Live Dashboard: lifecycle hooks publish events onto an Async::Queue; a consumer task IN THE SAME REACTOR renders the... | | `namespace_cartographer.rb` | The Namespace Cartographer: maps a gem's constant tree and audits every file against the constant Zeitwerk expects it to... | +| `one_file_api.rb` | The One-File API: an endpoint is a contract wearing HTTP. Declare the capability once and the rest is derived - the 422s... | | `perf_diff.rb` | The Perf Diff: run the plan before and after a change, diff per-task durations, and flag regressions - with the one qual... | | `perf_history.rb` | Perf History: last release's run left a journal; this release's run is compared against it. No synthetic baseline, no sa... | | `performance_detective.rb` | The Performance Detective: one task per Ruby file in lib/, fanned out through the orchestrator, each dissecting a file f... | | `plan_diagram.rb` | The Plan Diagrammer: any orchestrator's graph, emitted as Mermaid - paste it into a README, GitHub renders it, and the d... | +| `plan_forest.rb` | The Plan Forest: your graph drawn as a forest - roots at the soil, leaves in the canopy, every task planted at its depth... | | `plan_fortune.rb` | The Plan Fortune Teller: reads your graph's palm - depth, fan-in, roots, breadth - and tells its fortune. Every fortune ... | | `plan_gantt.rb` | The Plan Gantt: lifecycle hooks timestamp every task, then the run is rendered as an ASCII timeline - where your wall cl... | +| `plan_merge.rb` | The Plan Merge: base, ours, theirs - a three-way merge of plan wire formats. Independent changes combine; the same edge ... | | `plan_roundtrip.rb` | The Round Trip: serialize a plan's graph to JSON, rebuild a fresh orchestrator from the JSON, and prove the rebuilt topo... | | `plan_structural_diff.rb` | The Structural Diff: two versions of a plan's wire format, diffed as TOPOLOGY - tasks added and removed, edges rewired, ... | | `plan_tour.rb` | The Plan Tour: hand any orchestrator to the guide and it narrates the plan as prose - first this, then that, meanwhile t... | +| `polite_form.rb` | The Polite Form: a contract usually speaks AFTER you fail - a 422, a stack of violations. This assistant makes it speak ... | +| `projection_agreement.rb` | The Projection Agreement Prover: relation rules now render twice - the validator enforces them in Ruby, and to_json_sche... | | `quota_keeper.rb` | The Quota Keeper: the same 20 requests through two different laws. A concurrency ceiling ("3 in flight") models connecti... | | `readme_verifier.rb` | The README Verifier: every ruby code fence in the README is a promise. This extracts them all, syntax-checks each with P... | | `refactor_receipts.rb` | Refactor Receipts: the god-join plan from the graph critic, improved in two small steps - with a receipt after each one.... | | `refactoring_dojo.rb` | The Refactoring Dojo: a student submits a method, three critic agents review it from three distinct perspectives, and th... | +| `relation_diff.rb` | The Relation Diff: round 8's semver advisor classified declaration changes but had to shrug at rules - lambdas can't be ... | +| `relation_prober.rb` | The Relation Prober: relation-typed rules are new, and new predicates deserve hostility. Each relation is probed with ed... | | `renga_circle.rb` | A renga circle: three poet agents compose a linked-verse poem, each verse responding to the one before it. The dependenc... | +| `resize_torture.rb` | The Resize Torture Test: a feature that changes a limiter's ceiling while fibers are waiting on it had better say exactl... | +| `retry_budget.rb` | The Retry Budget: a retry storm is a self-inflicted DDoS - every job politely retrying 3x turns one outage into four. Re... | | `rule_prober.rb` | The Rule Prober: structured rules declare which fields they read - so now that claim can be AUDITED. For each rule, pert... | +| `rule_shapes.rb` | Rule Shapes: the same policy - "express shipments need a customs code" - written three ways: a lambda, a structured chec... | | `schema_advisor.rb` | The Schema Advisor: give it a schema and a query log, get back the advisories a careful DBA would write - each rule its ... | | `setup_doctor.rb` | The Setup Doctor: every onboarding wiki page is a bug. This runs the checks a README asks a new hire to do by hand - rub... | | `shared_rate_limit.rb` | The Shared Rate Limit: two plans run concurrently in one reactor, but the API key they share allows only 3 requests in f... | @@ -63,6 +90,9 @@ not this file. | `state_machine.rb` | The Contract State Machine: each transition is a capability whose guard is not an if-statement but an enum predicate on ... | | `telephone_game.rb` | The telephone game: a rumor passes through five villagers, each of whom hears the previous version through the orchestra... | | `three_shapes.rb` | Three Shapes: the same six units of work arranged three ways - a chain, a star, and staged joins - then measured and cri... | +| `throughput_knee.rb` | The Throughput Knee: sweep one limiter's ceiling from 1 to 8 against an upstream that quietly serializes above 4, and me... | | `ticket_screener.rb` | A HEY-style ticket screener: every inbound support ticket flows through screen -> categorize -> draft, all tickets in pa... | +| `traffic_dial.rb` | The Traffic Dial: a canary rollout as one knob. New code starts at one lane of traffic; every healthy stage turns the di... | | `typed_pipeline.rb` | A typed ETL pipeline: extract -> transform -> load, each stage a capability with a declared contract, composed into one ... | +| `variance_detective.rb` | The Variance Detective: ten journaled runs of the same plan, then a hunt for the task whose p90/p50 ratio betrays it. Av... | | `weekly_checkin.rb` | The Weekly Check-in: "what did you work on this week?" answered by the journal instead of by memory. Runs a few days of ... | diff --git a/examples/adaptive_throttle.rb b/examples/adaptive_throttle.rb new file mode 100644 index 0000000..4ea9680 --- /dev/null +++ b/examples/adaptive_throttle.rb @@ -0,0 +1,82 @@ +# frozen_string_literal: true + +# The Adaptive Throttle: nobody TELLS you an upstream's capacity - you +# discover it. An AIMD controller (TCP's algorithm) probes upward one +# lane at a time and halves on congestion, converging on the capacity +# the provider never documented. Watch the target find the truth. +# +# bundle exec ruby examples/adaptive_throttle.rb +# +# Runs offline; the upstream secretly handles 3 concurrent calls well. + +require_relative "../lib/agentic" +require "async" + +SECRET_CAPACITY = 3 +BASE_LATENCY = 0.02 +BATCHES = 12 +BATCH_SIZE = 6 + +# The upstream: fast until you exceed its capacity, then it degrades +in_flight = 0 +upstream = lambda do + in_flight += 1 + overload = [in_flight - SECRET_CAPACITY, 0].max + sleep(BASE_LATENCY * (1 + overload * 1.5)) + in_flight -= 1 +end + +# AIMD: additive increase, multiplicative decrease - steering ONE live +# limiter via resize, the same object the clients would share +target = 1 +limiter = Agentic::RateLimit.new(target) +history = [] +congestion_threshold = BASE_LATENCY * 1.6 + +puts "ADAPTIVE THROTTLE (upstream capacity: undisclosed; AIMD will find it)" +puts +puts format(" %-7s %-8s %-10s %-24s %s", "batch", "target", "p50", "", "action") + +Sync do + BATCHES.times do |batch| + limiter.resize(target) + latencies = [] + + BATCH_SIZE.times.map { + Async do + limiter.acquire do + started = Process.clock_gettime(Process::CLOCK_MONOTONIC) + upstream.call + latencies << Process.clock_gettime(Process::CLOCK_MONOTONIC) - started + end + end + }.each(&:wait) + + p50 = latencies.sort[latencies.size / 2] + congested = p50 > congestion_threshold + history << target + + action = if congested + new_target = [target / 2, 1].max + verdict = "congested -> halve to #{new_target}" + target = new_target + verdict + else + target += 1 + "healthy -> probe up to #{target}" + end + + puts format(" %-7d %-8d %6.1fms %-24s %s", + batch + 1, history.last, p50 * 1000, "#" * (history.last * 3), action) + end +end + +puts +settled = history.last(6) +puts format(" the controller oscillates around %.1f lanes - the upstream's", settled.sum.to_f / settled.size) +puts " secret capacity is #{SECRET_CAPACITY}. AIMD never saw that constant; it derived" +puts " it from latency alone, and it will re-derive it when the upstream" +puts " changes. static concurrency limits are a guess frozen at deploy" +puts " time; adaptive ones are a measurement that never stops. and the" +puts " controller steers ONE live RateLimit via resize - every client" +puts " sharing that limiter inherits each correction, mid-flight." diff --git a/examples/batch_import.rb b/examples/batch_import.rb new file mode 100644 index 0000000..1a80702 --- /dev/null +++ b/examples/batch_import.rb @@ -0,0 +1,89 @@ +# frozen_string_literal: true + +# The Batch Import: 500 rows of the kind of data people actually +# upload - typos, header drift, impossible combinations - run through +# one contract. Good rows proceed; bad rows land in a REJECT FILE +# with the field, the reason, and the rule that caught them. An +# importer that raises on row 37 is a tool for importing 36 rows. +# +# bundle exec ruby examples/batch_import.rb +# +# Runs offline; the dirty data is seeded and repeatable. + +require_relative "../lib/agentic" +require "json" + +CONTRACT = Agentic::CapabilitySpecification.new( + name: "import_shipment", description: "One row of the shipments upload", version: "1.0.0", + inputs: { + mode: {type: "string", required: true, enum: %w[air sea road]}, + weight: {type: "number", required: true, min: 1, max: 5_000}, + volume: {type: "number", min: 0}, + express: {type: "boolean"}, + customs_code: {type: "string"} + }, + rules: { + fits: {relation: :sum_lte, fields: [:weight, :volume], limit: 6_000}, + customs: {relation: :requires, fields: [:express, :customs_code]} + } +) + +# 500 rows, seeded: roughly three-quarters clean, the rest wrong in +# the ways uploads actually are +rng = Random.new(20_260_707) +ROWS = 500.times.map do |i| + row = { + mode: %w[air sea road].sample(random: rng), + weight: rng.rand(1..4_000), + volume: rng.rand(0..1_500) + } + row[:express] = true if rng.rand < 0.25 + row[:customs_code] = "HS-#{rng.rand(100)}" if row[:express] && rng.rand < 0.7 + case rng.rand + when 0..0.03 then row[:mode] = "trian" # typo + when 0.03..0.06 then row[:weight] = 0 # zero weight + when 0.06..0.09 then row[:weight] = rng.rand(5_001..9_000) # too heavy + when 0.09..0.12 then row[:volume] = rng.rand(4_000..8_000) # breaks the sum rule + when 0.12..0.14 then row.delete(:mode) # header drift + end + row +end + +validator = Agentic::CapabilityValidator.new(CONTRACT) +accepted = [] +rejects = [] + +started = Process.clock_gettime(Process::CLOCK_MONOTONIC) +ROWS.each_with_index do |row, index| + validator.validate_inputs!(row) + accepted << row +rescue Agentic::Errors::ValidationError => e + reasons = e.violations.except(:base).map { |field, msgs| "#{field}: #{msgs.first}" } + reasons += e.rule_violations.map { |v| "#{v[:rule]}: #{v[:message]}" } + rejects << {line: index + 2, reasons: reasons} # +2: 1-based plus header row +end +elapsed = Process.clock_gettime(Process::CLOCK_MONOTONIC) - started + +puts "BATCH IMPORT (#{ROWS.size} rows through one contract in #{(elapsed * 1000).round}ms)" +puts +puts " accepted: #{accepted.size} rejected: #{rejects.size}" +puts + +by_reason = rejects.flat_map { |r| r[:reasons].map { |reason| reason[/\A[^:]+/] } }.tally +puts " reject file, summarized by cause:" +by_reason.sort_by { |_, count| -count }.each do |cause, count| + puts format(" %-14s %-3d %s", cause, count, "#" * count) +end +puts +puts " first three lines of the reject file (the thing support actually opens):" +rejects.first(3).each do |reject| + puts " line #{reject[:line]}: #{reject[:reasons].join("; ")}" +end +puts +puts " #{ROWS.size} rows cost #{(elapsed * 1000).round}ms of validation - #{format("%.2f", elapsed / ROWS.size * 1_000_000)}us a row - and" +puts " every rejection names its line, its field, and its rule, including" +puts " the cross-field ones (\"fits\", \"customs\") no per-column check" +puts " catches. two design rules for importers: never raise on row 37" +puts " (collect, don't crash), and never write \"invalid row\" (a reject" +puts " file without reasons is a support ticket generator). the contract" +puts " supplied both for free." diff --git a/examples/cancel_drill.rb b/examples/cancel_drill.rb new file mode 100644 index 0000000..d1f649c --- /dev/null +++ b/examples/cancel_drill.rb @@ -0,0 +1,110 @@ +# frozen_string_literal: true + +# The Cancel Drill: structured concurrency's core promise is that +# cancellation is PROMPT - stop means stop, not "finish everything +# and then agree you'd stopped". Three drills measure what each +# cancel path actually delivers: the surgical ones keep the promise; +# the plan-wide one, under a joined reactor, turns out to be +# bookkeeping wearing a stop sign. +# +# bundle exec ruby examples/cancel_drill.rb +# +# Runs offline; every claim below is a measurement. + +require_relative "../lib/agentic" +require "async" + +Agentic.logger.level = :fatal + +JOB = 0.1 + +def build(count, agent_runs) + orchestrator = Agentic::PlanOrchestrator.new(concurrency_limit: 2) + epoch = Process.clock_gettime(Process::CLOCK_MONOTONIC) + tasks = count.times.map { |i| + Agentic::Task.new(description: "job#{i}", agent_spec: {"name" => "w", "instructions" => "w"}) + } + tasks.each do |task| + orchestrator.add_task(task, agent: ->(_t) { + agent_runs << [task.description, Process.clock_gettime(Process::CLOCK_MONOTONIC) - epoch] + sleep(JOB) + :ok + }) + end + [orchestrator, tasks] +end + +def states(orchestrator) + orchestrator.instance_variable_get(:@execution_state).transform_values(&:size) + .reject { |_, v| v.zero? }.map { |k, v| "#{v} #{k}" }.join(", ") +end + +puts "CANCEL DRILL (6 jobs of #{(JOB * 1000).to_i}ms, 2 lanes; full plan = 300ms)" +puts + +# --- drill 1: surgical cancel of one IN-FLIGHT task --------------------------- +runs = [] +orchestrator, tasks = build(6, runs) +elapsed = nil +Sync do + runner = Async { orchestrator.execute_plan } + Async do + sleep(0.03) # job0 and job1 are mid-sleep + orchestrator.cancel_task(tasks[0].id) + end + started = Process.clock_gettime(Process::CLOCK_MONOTONIC) + runner.wait + elapsed = Process.clock_gettime(Process::CLOCK_MONOTONIC) - started +end +third_start = (runs[2][1] * 1000).round +puts " drill 1 - cancel ONE in-flight task (at 30ms):" +puts " #{states(orchestrator)}; plan finished in #{(elapsed * 1000).round}ms" +puts " the proof the lane was freed is WHEN the next job started:" +puts " #{runs[2][0]} began at #{third_start}ms - on the canceled fiber's lane," +puts " immediately - not at 100ms when that job would have finished." +puts + +# --- drill 2: surgical cancel of one PENDING task ------------------------------ +runs2 = [] +orchestrator2, tasks2 = build(6, runs2) +Sync do + runner = Async { orchestrator2.execute_plan } + Async do + sleep(0.03) + orchestrator2.cancel_task(tasks2[5].id) # still queued behind the lanes + end + runner.wait +end +puts " drill 2 - cancel ONE pending task:" +puts " #{states(orchestrator2)}; agents actually ran: #{runs2.size}/6" +puts " the canceled job never started and never billed. queued work" +puts " canceled is money returned." +puts + +# --- drill 3: cancel_plan mid-flight ------------------------------------------- +runs3 = [] +orchestrator3, = build(6, runs3) +flip_ms = wall = nil +Sync do + started = Process.clock_gettime(Process::CLOCK_MONOTONIC) + runner = Async { orchestrator3.execute_plan } + Async do + sleep(0.03) + orchestrator3.cancel_plan + flip_ms = ((Process.clock_gettime(Process::CLOCK_MONOTONIC) - started) * 1000).round + end + runner.wait + wall = Process.clock_gettime(Process::CLOCK_MONOTONIC) - started +end +puts " drill 3 - cancel_plan at 30ms:" +puts " status flipped to :canceled by #{flip_ms}ms - and then the plan ran" +puts " #{(wall * 1000).round}ms anyway, with #{runs3.size}/6 agents executing (#{states(orchestrator3)})." +puts +puts " the drill's verdict: task-level cancel keeps the structured-" +puts " concurrency promise - stop is prompt, lanes are freed, unstarted" +puts " work stays unstarted. plan-level cancel under a joined reactor" +puts " is bookkeeping wearing a stop sign: every task reports :canceled" +puts " while every agent runs to completion and bills you, results" +puts " discarded. that's the worst trade available - full cost, zero" +puts " product. filed as the round-11 ask: cancel_plan must stop the" +puts " scheduler and the in-flight fibers, not just restate their status." diff --git a/examples/circuit_breaker.rb b/examples/circuit_breaker.rb new file mode 100644 index 0000000..67e4a9d --- /dev/null +++ b/examples/circuit_breaker.rb @@ -0,0 +1,137 @@ +# frozen_string_literal: true + +# The Circuit Breaker: when an upstream is down, the cheapest request +# is the one you don't send. The breaker trips after 3 consecutive +# retryable failures (or ONE non-retryable - no auth error deserves a +# second strike), fast-fails while open, and probes with a single +# request before closing again. Every skipped call is money. +# +# bundle exec ruby examples/circuit_breaker.rb +# +# Runs offline; act one is a 503 outage, act two a revoked key. + +require_relative "../lib/agentic" +require "tmpdir" + +Agentic.logger.level = :fatal + +# A breaker is a tiny state machine: closed -> open -> half_open -> closed +class Breaker + TRIP_AFTER = 3 + COOLDOWN_TICKS = 4 + + attr_reader :state, :skipped + + def initialize + @state = :closed + @strikes = 0 + @cooldown = 0 + @skipped = 0 + end + + def allow? + case @state + when :closed then true + when :open + @cooldown -= 1 + if @cooldown <= 0 + @state = :half_open + true # the probe + else + @skipped += 1 + false + end + when :half_open then true + end + end + + def record_success + @state = :closed + @strikes = 0 + end + + # The framework's nil convention (TaskFailure#hopeless?): only an + # EXPLICIT false verdict trips instantly. An error that expressed no + # opinion gets a strike - suspicion, not a death sentence. + def record_failure(verdict) + hopeless = verdict == false + @strikes += hopeless ? TRIP_AFTER : 1 + return unless @strikes >= TRIP_AFTER || @state == :half_open + + @state = :open + @strikes = 0 + @cooldown = COOLDOWN_TICKS + end +end + +def run_scenario(name, ticks, journal_path, &upstream) + File.delete(journal_path) if File.exist?(journal_path) + journal = Agentic::ExecutionJournal.new(path: journal_path) + breaker = Breaker.new + + puts " #{name}" + puts format(" %-6s %-11s %s", "tick", "breaker", "what happened") + + ticks.times do |tick| + unless breaker.allow? + puts format(" %-6d %-11s call SKIPPED - not sent, not billed, not waited on", tick, breaker.state) + next + end + + probe = breaker.state == :half_open + orchestrator = Agentic::PlanOrchestrator.new( + lifecycle_hooks: journal.lifecycle_hooks, + retry_policy: {max_retries: 0, retryable_errors: []} + ) + orchestrator.add_task(Agentic::Task.new( + description: "call:#{tick}", agent_spec: {"name" => "caller", "instructions" => "call"} + ), agent: ->(_t) { upstream.call(tick) }) + result = orchestrator.execute_plan + + if result.successful? + breaker.record_success + puts format(" %-6d %-11s %s", tick, breaker.state, probe ? "probe SUCCEEDED - breaker closes" : "ok") + else + # The journal's write-time verdict feeds the breaker - the error + # already testified whether retrying could ever help + verdict = Agentic::ExecutionJournal.replay(path: journal_path).events + .reverse.find { |e| e[:event] == "task_failed" }[:retryable] + breaker.record_failure(verdict) + puts format(" %-6d %-11s failed (retryable: %s)%s", tick, breaker.state, verdict, + (breaker.state == :open) ? " - breaker TRIPS" : "") + end + end + puts + [Agentic::ExecutionJournal.replay(path: journal_path), breaker] +end + +puts "CIRCUIT BREAKER (trip after #{Breaker::TRIP_AFTER} strikes, cooldown #{Breaker::COOLDOWN_TICKS} ticks)" +puts + +# Act one: a transient outage - three strikes, trip, skip, probe, recover +OUTAGE = (4..9) +state, breaker = run_scenario("act one: 503s from tick 4 to 9", 16, + File.join(Dir.tmpdir, "agentic_breaker_1.jsonl")) do |tick| + raise Agentic::Errors::LlmServerError, "503 (tick #{tick})" if OUTAGE.cover?(tick) + + "ok" +end + +# Act two: a revoked key - ONE strike, because the error testified +# that no retry can ever help +state2, breaker2 = run_scenario("act two: key revoked at tick 2", 8, + File.join(Dir.tmpdir, "agentic_breaker_2.jsonl")) do |tick| + raise Agentic::Errors::LlmAuthenticationError, "401 key revoked" if tick >= 2 + + "ok" +end + +felt = state.events.count { |e| e[:event] == "task_failed" } +puts " act one's outage lasted #{OUTAGE.size} ticks but only #{felt} calls felt it: the" +puts " breaker ate the middle as #{breaker.skipped} skips (nothing sent, nothing billed," +puts " no timeout waited out) and spent one probe discovering the recovery." +puts " act two never reached three strikes - the 401's journaled verdict" +puts " (retryable: false) tripped the breaker on FIRST contact, and the" +puts " probe kept finding the same wall (#{state2.events.count { |e| e[:event] == "task_failed" }} real calls, #{breaker2.skipped} skipped)." +puts " strike counts are for errors that might pass; testimony that the" +puts " error can never pass deserves an instant trip." diff --git a/examples/contract_fixtures.rb b/examples/contract_fixtures.rb new file mode 100644 index 0000000..3bfc902 --- /dev/null +++ b/examples/contract_fixtures.rb @@ -0,0 +1,146 @@ +# frozen_string_literal: true + +# Contract Fixtures: example payloads in docs rot the day the contract +# changes. So don't write them - DERIVE them. This generator reads a +# capability's declarations and produces a minimal fixture (required +# keys only) and a maximal one (everything), then a referee proves +# every generated fixture passes its own contract, and that a mutated +# fixture still fails. Docs that compile, effectively. +# +# bundle exec ruby examples/contract_fixtures.rb +# +# Runs offline; exits 1 if the generator and validator disagree. + +require_relative "../lib/agentic" +require "json" + +SPECS = [ + Agentic::CapabilitySpecification.new( + name: "quote_shipping", description: "Quote a shipment", version: "2.0.0", + inputs: { + mode: {type: "string", required: true, enum: %w[air sea road]}, + weight: {type: "number", required: true, min: 1, max: 5_000}, + volume: {type: "number", min: 0, max: 5_000}, + customs_code: {type: "string"}, + express: {type: "boolean"}, + api_key: {type: "string"}, + oauth_token: {type: "string"} + }, + outputs: {price_cents: {type: "number", required: true}}, + rules: { + fits: {relation: :sum_lte, fields: [:weight, :volume], limit: 4_000}, + customs: {relation: :requires, fields: [:express, :customs_code]}, + one_auth: {relation: :mutually_exclusive, fields: [:api_key, :oauth_token]} + } + ), + Agentic::CapabilitySpecification.new( + name: "classify_ticket", description: "Route a support ticket", version: "1.1.0", + inputs: { + text: {type: "string", required: true, non_empty: true}, + urgency: {type: "number", min: 0, max: 10} + }, + outputs: {queue: {type: "string", required: true, enum: %w[billing tech general]}} + ) +].freeze + +# One value per declaration, derived - never invented +def value_for(key, decl) + return decl[:enum].first if decl[:enum] + + case decl[:type] + when "number" + low = decl[:min] || 0 + high = decl[:max] || 100 + low + (high - low) / 2 + when "boolean" then true + else "example-#{key}" + end +end + +# Relation-typed rules are data, so the generator can SATISFY them +# instead of hoping: scale sums under their limit, add what a present +# trigger requires, keep only the first of an exclusive group +def satisfy_relations(fixture, spec) + spec.rules.each_value do |definition| + next if definition.respond_to?(:call) || !definition[:relation] + + fields = definition[:fields] + case definition[:relation] + when :sum_lte + given = fields.select { |f| fixture[f].is_a?(Numeric) } + total = given.sum { |f| fixture[f] } + if total > definition[:limit] + given.each { |f| fixture[f] = definition[:limit] / given.size } + end + when :requires + trigger, *needed = fields + if !fixture[trigger].nil? + needed.each { |f| fixture[f] ||= value_for(f, spec.inputs[f]) } + end + when :mutually_exclusive + fields.select { |f| fixture.key?(f) }.drop(1).each { |f| fixture.delete(f) } + end + end + fixture +end + +def fixtures_for(spec) + required = spec.inputs.select { |_, decl| decl[:required] } + { + "minimal" => satisfy_relations(required.to_h { |key, decl| [key, value_for(key, decl)] }, spec), + "maximal" => satisfy_relations(spec.inputs.to_h { |key, decl| [key, value_for(key, decl)] }, spec) + } +end + +failures = 0 +puts "CONTRACT FIXTURES (derived from declarations, then proved)" + +SPECS.each do |spec| + validator = Agentic::CapabilityValidator.new(spec) + puts + puts " #{spec.name} v#{spec.version}:" + + fixtures_for(spec).each do |flavor, fixture| + verdict = begin + validator.validate_inputs!(fixture) + "valid" + rescue Agentic::Errors::ValidationError => e + failures += 1 + "REJECTED BY OWN CONTRACT: #{e.message}" + end + puts format(" %-8s %-60s %s", flavor, JSON.generate(fixture), verdict) + end + + # The referee's teeth: a mutant fixture (first required key removed) + # must still FAIL - a validator that accepts everything would make + # the proofs above worthless + mutant = fixtures_for(spec)["minimal"].dup + removed = mutant.keys.first + mutant.delete(removed) + begin + validator.validate_inputs!(mutant) + failures += 1 + puts format(" %-8s dropped :%-20s ACCEPTED - validator has no teeth", "mutant", removed) + rescue Agentic::Errors::ValidationError + puts format(" %-8s dropped :%-20s rejected, as it must be", "mutant", removed) + end +end + +puts +if failures.zero? + puts " every derived fixture passed its own contract, and every mutant" + puts " failed. paste these into your README - when the contract changes," + puts " rerun and they change with it. handwritten examples are promises;" + puts " derived ones are consequences." + puts + puts " and the round-9 blind spot has closed for the declarable" + puts " majority: relation-typed rules (sum_lte, requires," + puts " mutually_exclusive) are data, so the generator SATISFIED them -" + puts " scaled the weights under the limit, added what express required," + puts " kept one credential of the exclusive pair - and the validator," + puts " which now enforces relations too, countersigned the result." + puts " lambdas remain for the exotic tail, and remain opaque." +else + puts " #{failures} DISAGREEMENT(S) between generator and validator." +end +exit(failures.zero? ? 0 : 1) diff --git a/examples/contract_overhead.rb b/examples/contract_overhead.rb new file mode 100644 index 0000000..419b6bc --- /dev/null +++ b/examples/contract_overhead.rb @@ -0,0 +1,77 @@ +# frozen_string_literal: true + +# The Contract Overhead Bench: "should we validate every call?" is a +# performance question, so answer it with a measurement instead of a +# vibe. Benchmarks the validator across contract sizes and rule +# counts, then prices it against the thing it protects - because +# overhead is a fraction, and everyone keeps quoting the numerator. +# +# bundle exec ruby examples/contract_overhead.rb +# +# Runs offline; times are real, the LLM latency is the industry's. + +require_relative "../lib/agentic" + +ITERATIONS = 2_000 +LLM_CALL_MS = 800.0 # a conservative round-trip for a real model call + +def bench(iterations) + # Warm up, then measure - the first call pays dry-schema compilation + yield + started = Process.clock_gettime(Process::CLOCK_MONOTONIC) + iterations.times { yield } + (Process.clock_gettime(Process::CLOCK_MONOTONIC) - started) / iterations * 1000 +end + +def spec_with(keys:, relations:) + inputs = keys.times.to_h { |i| [:"field_#{i}", {type: "number", required: true, min: 0, max: 1_000}] } + rules = relations.times.to_h { |i| + [:"rule_#{i}", {relation: :sum_lte, fields: [:"field_#{i}", :"field_#{(i + 1) % keys}"], limit: 2_000}] + } + Agentic::CapabilitySpecification.new( + name: "bench", description: "bench", version: "1.0.0", inputs: inputs, rules: rules + ) +end + +puts "CONTRACT OVERHEAD BENCH (#{ITERATIONS} validations per row)" +puts +puts format(" %-26s %-12s %s", "contract", "per call", "share of an #{LLM_CALL_MS.to_i}ms LLM call") + +rows = [ + ["3 keys, no rules", spec_with(keys: 3, relations: 0), {field_0: 1, field_1: 2, field_2: 3}], + ["10 keys, no rules", spec_with(keys: 10, relations: 0), 10.times.to_h { |i| [:"field_#{i}", i] }], + ["10 keys, 5 relations", spec_with(keys: 10, relations: 5), 10.times.to_h { |i| [:"field_#{i}", i] }], + ["30 keys, 15 relations", spec_with(keys: 30, relations: 15), 30.times.to_h { |i| [:"field_#{i}", i] }] +] + +results = rows.map do |label, spec, payload| + validator = Agentic::CapabilityValidator.new(spec) + ms = bench(ITERATIONS) { validator.validate_inputs!(payload) } + share = ms / LLM_CALL_MS * 100 + puts format(" %-26s %8.4fms %.4f%% %s", label, ms, share, "#" * [(share * 2000).round, 40].min) + [label, ms] +end + +# The failure path costs too - measure a rejection (exception + message building) +reject_spec = spec_with(keys: 10, relations: 5) +reject_validator = Agentic::CapabilityValidator.new(reject_spec) +bad_payload = 10.times.to_h { |i| [:"field_#{i}", 1_900] } # breaks every sum_lte +reject_ms = bench(500) { + begin + reject_validator.validate_inputs!(bad_payload) + rescue Agentic::Errors::ValidationError + nil + end +} +puts format(" %-26s %8.4fms (the expensive path: exception + %d rule reports)", "rejection, 5 rules broken", reject_ms, 5) + +puts +fastest, cheapest = results.first +biggest, priciest = results.last +puts " the whole table rounds to zero: the biggest contract here" +puts format(" (%s) costs %.4fms per call - %.5f%% of the LLM", biggest.downcase, priciest, priciest / LLM_CALL_MS * 100) +puts format(" round-trip it guards. even rejection, the slow path, is %.2fms.", reject_ms) +puts " \"we skip validation for performance\" saves the price of a" +puts format(" rounding error (%s: %.4fms) to risk shipping a malformed", fastest.downcase, cheapest) +puts " prompt to an #{LLM_CALL_MS.to_i}ms call that BILLS you for the mistake." +puts " validate both doors. the meter says you can afford it." diff --git a/examples/contract_semver.rb b/examples/contract_semver.rb new file mode 100644 index 0000000..94a73ca --- /dev/null +++ b/examples/contract_semver.rb @@ -0,0 +1,94 @@ +# frozen_string_literal: true + +# The Contract Semver Advisor: two versions of a capability's contract, +# every change classified as breaking or compatible - FROM THE CALLER'S +# and THE CONSUMER'S seats, which disagree about what "breaking" means. +# The verdict is the version bump you owe your users. +# +# bundle exec ruby examples/contract_semver.rb +# +# Runs offline; v2 contains one of every interesting change. + +require_relative "../lib/agentic" + +V1 = Agentic::CapabilitySpecification.new( + name: "quote_shipping", description: "Quote a shipment", version: "1.4.0", + inputs: { + mode: {type: "string", required: true, enum: %w[air sea]}, + weight: {type: "number", required: true, min: 1, max: 10_000}, + notes: {type: "string"} + }, + outputs: { + price_cents: {type: "number", required: true}, + carrier: {type: "string", required: true} + } +) + +V2 = Agentic::CapabilitySpecification.new( + name: "quote_shipping", description: "Quote a shipment", version: "?", + inputs: { + mode: {type: "string", required: true, enum: %w[air sea road]}, # enum widened + weight: {type: "number", required: true, min: 1, max: 5_000}, # max tightened + customs_code: {type: "string", required: true}, # new required input + notes: {type: "string"} + }, + outputs: { + price_cents: {type: "number", required: true}, + eta_days: {type: "number", required: true} # new output + # carrier: removed + } +) + +def classify_inputs(v1, v2) + changes = [] + v2.inputs.each do |key, decl| + old = v1.inputs[key] + if old.nil? + changes << [decl[:required] ? :breaking : :compatible, + "input :#{key} added#{decl[:required] ? " as REQUIRED - existing callers don't send it" : " (optional)"}"] + next + end + changes << [:breaking, "input :#{key} type changed #{old[:type]} -> #{decl[:type]}"] if old[:type] != decl[:type] + changes << [:breaking, "input :#{key} became required"] if decl[:required] && !old[:required] + if old[:enum] && decl[:enum] + changes << [:compatible, "input :#{key} enum widened (#{(decl[:enum] - old[:enum]).join(", ")})"] if (old[:enum] - decl[:enum]).empty? && decl[:enum] != old[:enum] + changes << [:breaking, "input :#{key} enum narrowed (removed #{(old[:enum] - decl[:enum]).join(", ")})"] unless (old[:enum] - decl[:enum]).empty? + end + changes << [:breaking, "input :#{key} max tightened #{old[:max]} -> #{decl[:max]} - previously legal calls now rejected"] if old[:max] && decl[:max] && decl[:max] < old[:max] + changes << [:breaking, "input :#{key} min tightened #{old[:min]} -> #{decl[:min]}"] if old[:min] && decl[:min] && decl[:min] > old[:min] + end + (v1.inputs.keys - v2.inputs.keys).each do |key| + changes << [:compatible, "input :#{key} no longer declared (extra keys were always permitted)"] + end + changes +end + +def classify_outputs(v1, v2) + changes = [] + (v1.outputs.keys - v2.outputs.keys).each do |key| + changes << [:breaking, "output :#{key} removed - consumers reading it get nil"] + end + (v2.outputs.keys - v1.outputs.keys).each do |key| + changes << [:compatible, "output :#{key} added (consumers ignore unknown keys)"] + end + changes +end + +changes = classify_inputs(V1, V2) + classify_outputs(V1, V2) +breaking = changes.count { |kind, _| kind == :breaking } + +puts "CONTRACT SEMVER ADVISOR: #{V1.name} v#{V1.version} -> v?" +puts +changes.sort_by { |kind, _| (kind == :breaking) ? 0 : 1 }.each do |kind, message| + puts format(" %-10s %s", kind.to_s.upcase, message) +end + +major, minor, = V1.version.split(".").map(&:to_i) +suggested = breaking.positive? ? "#{major + 1}.0.0" : "#{major}.#{minor + 1}.0" +puts +puts " verdict: #{breaking} breaking change(s) -> ship as v#{suggested}" +puts +puts " note the asymmetry: INPUTS break when tightened (callers rejected)," +puts " OUTPUTS break when narrowed (consumers starved). the same edit is" +puts " breaking on one side and compatible on the other - semver for" +puts " contracts is a two-seat calculation, and both seats are customers." diff --git a/examples/dead_letter_office.rb b/examples/dead_letter_office.rb new file mode 100644 index 0000000..bfdfc12 --- /dev/null +++ b/examples/dead_letter_office.rb @@ -0,0 +1,93 @@ +# frozen_string_literal: true + +# The Dead Letter Office: three days of journaled runs, every failure +# collected and triaged by what the errors said about themselves - +# retryable failures go on the requeue manifest, non-retryable ones +# get parked with a reason. Nobody re-sends a letter addressed to a +# revoked mailbox. +# +# bundle exec ruby examples/dead_letter_office.rb +# +# Runs offline; failures are scripted across runs. + +require_relative "../lib/agentic" +require "tmpdir" + +Agentic.logger.level = :fatal + +JOURNAL = File.join(Dir.tmpdir, "agentic_dlo.journal.jsonl") +File.delete(JOURNAL) if File.exist?(JOURNAL) +journal = Agentic::ExecutionJournal.new(path: JOURNAL) + +RUNS = [ + {"sync:crm" => Agentic::Errors::LlmTimeoutError.new("read timeout"), + "sync:billing" => Agentic::Errors::LlmRateLimitError.new("429, slow down"), + "sync:tickets" => nil}, + {"sync:crm" => nil, # recovers on its own + "sync:billing" => Agentic::Errors::LlmRateLimitError.new("429, still angry"), + "sync:warehouse" => Agentic::Errors::LlmAuthenticationError.new("401 key revoked")}, + {"sync:tickets" => Agentic::Errors::LlmServerError.new("502 from upstream"), + "sync:warehouse" => Agentic::Errors::LlmAuthenticationError.new("401 key revoked")} +].freeze + +RUNS.each do |jobs| + orchestrator = Agentic::PlanOrchestrator.new( + concurrency_limit: 2, lifecycle_hooks: journal.lifecycle_hooks, + retry_policy: {max_retries: 0, retryable_errors: []} + ) + jobs.each do |name, error| + orchestrator.add_task(Agentic::Task.new( + description: name, agent_spec: {"name" => name, "instructions" => "sync"}, + payload: error + ), agent: ->(t) { + sleep(0.005) + raise t.payload if t.payload + + :ok + }) + end + orchestrator.execute_plan +end + +# --- the office: triage from the journal -------------------------------------- +state = Agentic::ExecutionJournal.replay(path: JOURNAL) + +# A letter is DEAD only if its most recent attempt failed +latest = {} +state.events.each do |event| + next unless %w[task_succeeded task_failed].include?(event[:event]) + + latest[event[:description]] = event +end +dead = latest.values.select { |e| e[:event] == "task_failed" } + +# Each failure's retryability was journaled AT THE MOMENT it happened +# (from the error's own retryable? verdict) - no read-time table to +# drift out of date when the taxonomy renames +requeue, parked = dead.partition { |e| e[:retryable] } +attempts = state.events.select { |e| e[:event] == "task_failed" }.group_by { |e| e[:description] } + +puts "DEAD LETTER OFFICE (#{RUNS.size} journaled runs)" +puts +puts " REQUEUE MANIFEST (transient failures - safe to retry):" +requeue.each do |letter| + puts format(" %-16s %-42s %d failed attempt(s) on record", + letter[:description], "#{letter[:error_type].split("::").last}: #{letter[:error]}", + attempts[letter[:description]].size) +end +puts +puts " PARKED (retrying will not help - a human must act):" +parked.each do |letter| + puts format(" %-16s %s", letter[:description], "#{letter[:error_type].split("::").last}: #{letter[:error]}") +end +puts +recovered = latest.values.select { |e| e[:event] == "task_succeeded" && attempts.key?(e[:description]) } +puts " recovered on their own (failed once, succeeded later - NOT dead):" +recovered.each { |e| puts " #{e[:description]}" } +puts +puts " the office triages by MOST RECENT attempt: sync:crm's old timeout" +puts " doesn't page anyone, and sync:tickets' early success doesn't" +puts " excuse its newer 502. a dead-letter queue that forgets recoveries" +puts " pages people for ghosts; one that forgets relapses buries real mail." +puts " and each verdict above came from the journal itself - retryability" +puts " was recorded when the error was fresh, not reconstructed later." diff --git a/examples/duck_agents.rb b/examples/duck_agents.rb new file mode 100644 index 0000000..c3e4b42 --- /dev/null +++ b/examples/duck_agents.rb @@ -0,0 +1,100 @@ +# frozen_string_literal: true + +# Duck Agents: the agent: seam asks one question - "can you be called +# with a task?" - and five differently-shaped objects all answer yes: +# a lambda, an instance, a Method, a curried proc, and a decorator +# wrapping any of the others. Nobody was asked what class they are. +# Ask for what you need, not for who someone is. +# +# bundle exec ruby examples/duck_agents.rb +# +# Runs offline; one plan, five ducks, one timing decorator. + +require_relative "../lib/agentic" + +Agentic.logger.level = :fatal + +# Duck 1: a lambda - the shape you reach for first +fetch = ->(task) { {records: 12, source: task.description} } + +# Duck 2: a plain object with #call - state and a real home for logic +class Deduper + def initialize + @seen = 0 + end + + def call(_task) + @seen += 1 + {unique: 9, dropped: 3, pass: @seen} + end +end + +# Duck 3: a Method object - module functions join the plan unwrapped +module Stats + def self.summarize(_task) + {mean: 4.2, max: 9} + end +end + +# Duck 4: a curried proc - configuration applied ahead of time, +# leaving exactly the one-argument shape the seam asks for +render = lambda { |format, _task| {rendered: true, format: format} } +render_html = render.curry["html"] + +# Duck 5: a decorator - wraps ANY of the above, because it only +# relies on the same one-message contract it provides +class Timed + def initialize(inner) + @inner = inner + end + + def call(task) + started = Process.clock_gettime(Process::CLOCK_MONOTONIC) + @inner.call(task).merge( + timed_ms: ((Process.clock_gettime(Process::CLOCK_MONOTONIC) - started) * 1000).round(2) + ) + end +end + +def task(name) + Agentic::Task.new(description: name, agent_spec: {"name" => name, "instructions" => "work"}) +end + +orchestrator = Agentic::PlanOrchestrator.new +fetch_task = task("fetch") +dedupe_task = task("dedupe") +stats_task = task("stats") +render_task = task("render") +audit_task = task("audit") + +orchestrator.add_task(fetch_task, agent: fetch) +orchestrator.add_task(dedupe_task, [fetch_task], agent: Deduper.new) +orchestrator.add_task(stats_task, [dedupe_task], agent: Stats.method(:summarize)) +orchestrator.add_task(render_task, [stats_task], agent: render_html) +orchestrator.add_task(audit_task, [render_task], agent: Timed.new(->(_t) { {audited: true} })) + +result = orchestrator.execute_plan + +DUCKS = { + "fetch" => "lambda", + "dedupe" => "instance with #call", + "stats" => "Method object", + "render" => "curried proc", + "audit" => "decorator around a lambda" +}.freeze + +puts "DUCK AGENTS (one seam, five shapes)" +puts +[fetch_task, dedupe_task, stats_task, render_task, audit_task].each do |t| + output = result.task_result(t.id).output + puts format(" %-8s %-26s -> %s", t.description, DUCKS[t.description], output) +end + +puts +puts " the seam asked one question: respond to call (or execute) with a" +puts " task. it never asked anyone's class, so anything shaped right" +puts " walks in: closures for the quick cases, instances when logic" +puts " deserves a home, Method objects when it already has one, curry" +puts " for pre-applied config, and decorators that stack because they" +puts " honor the same contract they consume. depend on messages and" +puts " every object ever written - and not yet written - is a plugin." diff --git a/examples/eval_scorers.rb b/examples/eval_scorers.rb new file mode 100644 index 0000000..62e8f27 --- /dev/null +++ b/examples/eval_scorers.rb @@ -0,0 +1,110 @@ +# frozen_string_literal: true + +# Eval Scorers: the same eval set scored four ways - exact match, +# keyword containment, numeric tolerance, and a judge rubric. Exact +# scoring drowns one real failure in wording noise; the right scorer +# per field reports exactly the failure that matters. The harness +# shape never changes - only the scorer column does. +# +# bundle exec ruby examples/eval_scorers.rb +# +# Runs offline; exits 1 because one capability blind spot is real. + +require_relative "../lib/agentic" + +SEVERITY = { + "damaged item" => 0.9, + "refund requested" => 0.8, + "account email update" => 0.2 +}.freeze + +spec = Agentic::CapabilitySpecification.new( + name: "summarize_ticket", description: "Summarize a support ticket", version: "1.0.0", + inputs: {text: {type: "string", required: true}}, + outputs: { + summary: {type: "string", required: true}, + priority: {type: "number", required: true, min: 0, max: 1} + } +) +Agentic.register_capability(spec, Agentic::CapabilityProvider.new(capability: spec, implementation: ->(i) { + text = i[:text].downcase + fragments = [] + fragments << "damaged item" if text.match?(/broken|damaged/) + fragments << "refund requested" if text.match?(/refund|money back/) + fragments << "account email update" if text.match?(/email/) + # the blind spot: no rule for crashes - those tickets read as general inquiries + summary = fragments.empty? ? "general inquiry" : "customer reports #{fragments.join(", ")}" + {summary: summary, priority: fragments.map { |f| SEVERITY[f] }.max || 0.3} +})) + +# --- the scorer seam: (expected, actual) -> score in 0.0..1.0 ------------------ +SCORERS = { + exact: ->(expected, actual) { (expected == actual) ? 1.0 : 0.0 }, + contains: ->(keywords, actual) { keywords.count { |k| actual.to_s.downcase.include?(k) }.fdiv(keywords.size) }, + tolerance: ->(spec, actual) { ((spec[:value] - actual).abs <= spec[:within]) ? 1.0 : 0.0 }, + judge: ->(rubric, actual) { rubric.call(actual) } +}.freeze +PASS_AT = 0.99 # judge scorers may grade partially; everything else is 0-or-1 + +NAMES_A_PROBLEM = ->(summary) { (summary.include?("general inquiry") ? 0.0 : 0.6) } +NAMES_AN_ACTION = ->(summary) { summary.match?(/request|update/) ? 0.4 : 0.0 } +RUBRIC = ->(summary) { NAMES_A_PROBLEM.call(summary) + NAMES_AN_ACTION.call(summary) } + +CASES = [ + {ticket: "My package arrived broken and I want my money back", + checks: [ + {field: :summary, scorer: :exact, expected: "Damaged item; refund requested"}, + {field: :summary, scorer: :contains, expected: %w[damaged refund]}, + {field: :priority, scorer: :tolerance, expected: {value: 0.9, within: 0.15}}, + {field: :summary, scorer: :judge, expected: RUBRIC} + ]}, + {ticket: "How do I change my email address?", + checks: [ + {field: :summary, scorer: :exact, expected: "customer reports account email update"}, + {field: :summary, scorer: :contains, expected: %w[email]}, + {field: :priority, scorer: :tolerance, expected: {value: 0.2, within: 0.1}} + ]}, + {ticket: "The app crashes every time I open settings and I lost work", + checks: [ + {field: :summary, scorer: :exact, expected: "Crash in settings; data loss"}, + {field: :summary, scorer: :contains, expected: %w[crash settings]}, + {field: :priority, scorer: :tolerance, expected: {value: 0.95, within: 0.1}}, + {field: :summary, scorer: :judge, expected: RUBRIC} + ]} +].freeze + +provider = Agentic::AgentCapabilityRegistry.instance.get_provider("summarize_ticket") +results = CASES.flat_map.with_index(1) do |kase, number| + output = provider.execute(text: kase[:ticket]) + kase[:checks].map do |check| + score = SCORERS.fetch(check[:scorer]).call(check[:expected], output[check[:field]]) + {case: number, scorer: check[:scorer], field: check[:field], score: score, pass: score >= PASS_AT} + end +end + +puts "EVAL SCORERS: one eval set, four ways to say \"good enough\"" +puts +CASES.each_with_index do |kase, index| + puts " case #{index + 1}: #{kase[:ticket].inspect}" + results.select { |r| r[:case] == index + 1 }.each do |r| + puts format(" %-10s on %-9s %s (%.2f)", r[:scorer], r[:field], r[:pass] ? "PASS" : "FAIL", r[:score]) + end + puts +end + +by_scorer = results.group_by { |r| r[:scorer] } +puts " scoreboard:" +by_scorer.each do |scorer, rows| + puts format(" %-10s %d/%d pass", scorer, rows.count { |r| r[:pass] }, rows.size) +end + +exact_fails = by_scorer[:exact].count { |r| !r[:pass] } +real_fails = results.reject { |r| r[:scorer] == :exact }.reject { |r| r[:pass] }.map { |r| r[:case] }.uniq +puts +puts " exact flagged #{exact_fails}/3 cases, but most of that is wording noise." +puts " the field-appropriate scorers flagged only case #{real_fails.join(", ")} - the crash" +puts " ticket - and that failure is REAL: the capability has no rule for" +puts " crashes, so a data-loss ticket scores priority 0.3. same harness," +puts " same cases; the scorer column is what makes a failure mean something." + +exit(real_fails.any? ? 1 : 0) diff --git a/examples/failure_weather.rb b/examples/failure_weather.rb new file mode 100644 index 0000000..2a7f4ba --- /dev/null +++ b/examples/failure_weather.rb @@ -0,0 +1,104 @@ +# frozen_string_literal: true + +# The Failure Weather Report: a journal of three days, read as a +# forecast. Retryable failures are WEATHER - showers that pass on +# their own or with an umbrella. Non-retryable failures are CLIMATE - +# no amount of waiting fixes a drought; someone must dig a well. +# The journal now records which is which at the moment it rains. +# +# bundle exec ruby examples/failure_weather.rb +# +# Runs offline; three scripted days of mixed conditions. + +require_relative "../lib/agentic" +require "tmpdir" + +Agentic.logger.level = :fatal + +JOURNAL = File.join(Dir.tmpdir, "agentic_weather.journal.jsonl") +File.delete(JOURNAL) if File.exist?(JOURNAL) +journal = Agentic::ExecutionJournal.new(path: JOURNAL) + +DAYS = [ + {name: "Monday", + jobs: {"digest" => Agentic::Errors::LlmRateLimitError.new("429"), + "backup" => Agentic::Errors::LlmTimeoutError.new("slow disk"), + "invoice" => Agentic::Errors::LlmAuthenticationError.new("401 key expired"), + "greet" => nil}}, + {name: "Tuesday", + jobs: {"digest" => nil, # the shower passed + "backup" => Agentic::Errors::LlmTimeoutError.new("slow disk again"), + "invoice" => Agentic::Errors::LlmAuthenticationError.new("401 key expired"), + "greet" => nil}}, + {name: "Wednesday", + jobs: {"digest" => nil, + "backup" => nil, # cleared overnight + "invoice" => Agentic::Errors::LlmAuthenticationError.new("401 key expired"), + "greet" => nil}} +].freeze + +DAYS.each do |day| + orchestrator = Agentic::PlanOrchestrator.new( + lifecycle_hooks: journal.lifecycle_hooks, + retry_policy: {max_retries: 0, retryable_errors: []} + ) + day[:jobs].each do |name, error| + orchestrator.add_task(Agentic::Task.new( + description: name, agent_spec: {"name" => name, "instructions" => "run"}, + payload: error + ), agent: ->(t) { + raise t.payload if t.payload + + :ok + }) + end + orchestrator.execute_plan +end + +# --- the forecast desk --------------------------------------------------------- +state = Agentic::ExecutionJournal.replay(path: JOURNAL) +day_events = state.events.slice_when { |a, b| + a[:event] == "plan_completed" && b[:event] != "plan_completed" +}.to_a + +def sky(failed) + weather = failed.count { |e| e[:retryable] } + climate = failed.count { |e| e[:retryable] == false } + return "clear skies" if failed.empty? + return "storm damage (#{climate} structural)" if climate.positive? && weather.positive? + return "drought continues" if climate.positive? + + "passing showers (#{weather})" +end + +puts "FAILURE WEATHER REPORT (#{DAYS.size} journaled days)" +puts +day_events.each_with_index do |events, index| + failed = events.select { |e| e[:event] == "task_failed" } + puts format(" %-10s %-28s %s", DAYS[index][:name], sky(failed), + failed.map { |e| e[:description] }.join(", ")) +end +puts + +# Weather clears; climate persists. The distinction IS the journal's +# retryable verdict, recorded when each drop fell. +latest = {} +state.events.each do |e| + latest[e[:description]] = e if %w[task_failed task_succeeded].include?(e[:event]) +end +weather_jobs = latest.values.select { |e| e[:event] == "task_failed" && e[:retryable] } +climate_jobs = latest.values.select { |e| e[:event] == "task_failed" && e[:retryable] == false } +cleared = state.events.select { |e| e[:event] == "task_failed" }.map { |e| e[:description] }.uniq + .select { |d| latest[d][:event] == "task_succeeded" } + +puts " extended forecast:" +cleared.each { |d| puts " #{d}: rained earlier this week, clear now - weather does that" } +weather_jobs.each { |e| puts " #{e[:description]}: still raining, but it is rain - bring retries" } +climate_jobs.each { |e| puts " #{e[:description]}: this is not weather, it is climate - #{e[:error]}" } +puts +rain = state.events.select { |e| e[:event] == "task_failed" } +puts " #{rain.size} rainy events this week: #{rain.count { |e| e[:retryable] }} were weather (they passed, or will)," +puts " and #{rain.count { |e| e[:retryable] == false }} were the same drought, reported daily." +puts " no forecast fixes a drought: invoice's 401 has held for three days" +puts " and will hold forever, because keys do not expire back. the journal" +puts " told us which failures to wait out and which to dig a well for." diff --git a/examples/fair_share.rb b/examples/fair_share.rb new file mode 100644 index 0000000..e5d6ab3 --- /dev/null +++ b/examples/fair_share.rb @@ -0,0 +1,98 @@ +# frozen_string_literal: true + +# Fair Share: two tenants, one upstream. The global ceiling is fair to +# REQUESTS - first come, first served - but tenant A brings 6 workers +# and tenant B brings 2, so "fair to requests" quietly means "A gets +# triple". Per-tenant ceilings under the global door restore fairness +# to TENANTS; resize keeps the idle tenant's share from stranding. +# +# bundle exec ruby examples/fair_share.rb +# +# Runs offline; watch B's number - it tells the whole story. + +require_relative "../lib/agentic" +require "async" + +GLOBAL = 4 +JOB = 0.01 +PHASE = 0.24 +WORKERS = {a: 6, b: 2}.freeze # A is greedy; B just wants its two lanes + +global = Agentic::RateLimit.new(GLOBAL) +share_a = Agentic::RateLimit.new(2) +share_b = Agentic::RateLimit.new(2) +tenant_a = share_a.and(global) # own share first, then the shared door +tenant_b = share_b.and(global) + +served = Hash.new(0) + +# A tenant is N worker fibers, each pushing as hard as its limiter allows +def run_tenants(served, plan) + Sync do + plan.flat_map { |key, (limit, workers)| + workers.times.map { + Async do + deadline = Process.clock_gettime(Process::CLOCK_MONOTONIC) + PHASE + while Process.clock_gettime(Process::CLOCK_MONOTONIC) < deadline + limit.acquire { + sleep(JOB) + served[key] += 1 + } + end + end + } + }.each(&:wait) + end +end + +def phase(title, served, shares) + before = served.dup + yield + puts format(" %-36s A: %-4d B: %-4d (shares %s)", + title, served[:a] - before[:a], served[:b] - before[:b], shares) +end + +puts "FAIR SHARE (global ceiling #{GLOBAL}; A brings #{WORKERS[:a]} workers, B brings #{WORKERS[:b]})" +puts + +# Phase 1 - no shares: the door is fair to requests, so the tenant +# with more workers takes proportionally more. B wants 2 lanes' worth +# and gets half of it. +phase("no shares, one door for all", served, "-/-") do + run_tenants(served, {a: [global, WORKERS[:a]], b: [global, WORKERS[:b]]}) +end + +# Phase 2 - 2/2 shares under the door: B reaches its full demand no +# matter how many workers A hires +phase("2/2 shares, same greedy A", served, "2/2") do + run_tenants(served, {a: [tenant_a, WORKERS[:a]], b: [tenant_b, WORKERS[:b]]}) +end + +# Phase 3 - B goes idle; static shares strand B's lanes +phase("B idle, static 2/2 shares", served, "2/2") do + run_tenants(served, {a: [tenant_a, WORKERS[:a]], b: [tenant_b, 0]}) +end + +# Phase 4 - same idle B, but the spare share is lent to A, live +share_a.resize(4) +share_b.resize(1) +phase("B idle, shares rebalanced 4/1", served, "4/1") do + run_tenants(served, {a: [tenant_a, WORKERS[:a]], b: [tenant_b, 0]}) +end + +# Phase 5 - B returns; the share comes back, live +share_a.resize(2) +share_b.resize(2) +phase("B returns, shares back to 2/2", served, "2/2") do + run_tenants(served, {a: [tenant_a, WORKERS[:a]], b: [tenant_b, WORKERS[:b]]}) +end + +puts +puts " phase 1 is the quiet outage: nothing errored, nothing paged - B" +puts " simply got half its lanes because the door counts requests, not" +puts " tenants. phase 2 buys tenant-fairness by composition: own share" +puts " first, then the door. phase 3 is the tax static shares charge -" +puts " B's idle lanes served nobody - and phases 4-5 are the round-9" +puts " payoff: resize lends the idle share and takes it back, live," +puts " while the composition never changes shape. fairness is a policy;" +puts " make it an object and it becomes an adjustable one." diff --git a/examples/graph_invariants.rb b/examples/graph_invariants.rb new file mode 100644 index 0000000..c0eefe1 --- /dev/null +++ b/examples/graph_invariants.rb @@ -0,0 +1,137 @@ +# frozen_string_literal: true + +# The Graph Invariants Prover: the reflection API makes promises - +# order respects edges, roots have no dependencies, depth is the +# longest path, leaves feed nothing. Documentation asserts these; +# this referee PROVES them, across four plan shapes including a +# deliberate cycle. Exit 0 is a certificate, not a shrug. +# +# bundle exec ruby examples/graph_invariants.rb +# +# Runs offline; exits 1 if any invariant is violated. + +require_relative "../lib/agentic" + +def task(name) + Agentic::Task.new(description: name, agent_spec: {"name" => name, "instructions" => "work"}) +end + +def chain_plan + orchestrator = Agentic::PlanOrchestrator.new + a, b, c, d = %w[a b c d].map { |n| task(n) } + orchestrator.add_task(a) + orchestrator.add_task(b, [a]) + orchestrator.add_task(c, [b]) + orchestrator.add_task(d, [c]) + orchestrator +end + +def diamond_plan + orchestrator = Agentic::PlanOrchestrator.new + top, left, right, bottom = %w[top left right bottom].map { |n| task(n) } + orchestrator.add_task(top) + orchestrator.add_task(left, [top]) + orchestrator.add_task(right, [top]) + orchestrator.add_task(bottom, needs: {l: left, r: right}) + orchestrator +end + +def forest_plan + orchestrator = Agentic::PlanOrchestrator.new + trees = %w[oak elm ash].map { |n| task(n) } + trees.each { |t| orchestrator.add_task(t) } + crown = task("crown") + orchestrator.add_task(crown, trees) + lone = task("lone") + orchestrator.add_task(lone) + orchestrator +end + +def cyclic_plan + orchestrator = Agentic::PlanOrchestrator.new + x, y = %w[x y].map { |n| task(n) } + orchestrator.add_task(x, [y.id]) + orchestrator.add_task(y, [x]) + orchestrator +end + +# Each invariant is a lambda: graph in, list of violations out +INVARIANTS = { + "order is a permutation of the task set" => lambda { |g| + (g[:order].sort == g[:tasks].keys.sort) ? [] : ["order #{g[:order].size} ids, tasks #{g[:tasks].size}"] + }, + "order respects every edge (acyclic only)" => lambda { |g| + position = g[:order].each_with_index.to_h + g[:edges].reject { |e| position[e[:from]] < position[e[:to]] } + .map { |e| "edge #{e[:from]}->#{e[:to]} out of order" } + }, + "roots are exactly the tasks with no dependencies" => lambda { |g| + expected = g[:dependencies].select { |_, deps| deps.empty? }.keys + (g[:stats][:roots].sort == expected.sort) ? [] : ["roots mismatch"] + }, + "leaves are exactly the tasks nothing depends on" => lambda { |g| + fed = g[:dependencies].values.flatten + expected = g[:tasks].keys - fed + (g[:stats][:leaves].sort == expected.sort) ? [] : ["leaves mismatch"] + }, + "depth is 1 + max dependency depth (acyclic only)" => lambda { |g| + g[:tasks].keys.filter_map { |id| + deps = g[:dependencies][id] + expected = deps.empty? ? 1 : 1 + deps.map { |d| g[:stats][:depth][d] || 0 }.max + "depth[#{id}] = #{g[:stats][:depth][id]}, expected #{expected}" if g[:stats][:depth][id] != expected + } + }, + "max_depth and max_fan_in agree with their sources" => lambda { |g| + violations = [] + violations << "max_depth" if g[:stats][:max_depth] != (g[:stats][:depth].values.max || 0) + violations << "max_fan_in" if g[:stats][:max_fan_in] != (g[:dependencies].values.map(&:size).max || 0) + violations + }, + "every needs: label appears on its edge" => lambda { |g| + g[:needs].flat_map { |task_id, named| + named.filter_map { |label, dep_id| + edge = g[:edges].find { |e| e[:from] == dep_id && e[:to] == task_id } + "label #{label} missing on #{dep_id}->#{task_id}" if edge.nil? || edge[:label] != label + } + } + } +}.freeze + +PLANS = { + "chain (a->b->c->d)" => chain_plan, + "diamond (labeled join)" => diamond_plan, + "forest (3 trees + orphan)" => forest_plan, + "cycle (x<->y)" => cyclic_plan +}.freeze + +puts "GRAPH INVARIANTS PROVER (#{INVARIANTS.size} invariants x #{PLANS.size} plan shapes)" +puts +failures = 0 +PLANS.each do |plan_name, orchestrator| + graph = orchestrator.graph + cyclic = plan_name.include?("cycle") + puts " #{plan_name}:" + INVARIANTS.each do |invariant_name, check| + next if cyclic && invariant_name.include?("acyclic only") + + violations = check.call(graph) + failures += violations.size + status = violations.empty? ? "proved" : "VIOLATED: #{violations.join("; ")}" + puts format(" %-52s %s", invariant_name, status) + end + puts +end + +if failures.zero? + puts " #{INVARIANTS.size * PLANS.size - 2} proofs, 0 violations. two invariants excuse themselves" + puts " on the cycle - and finding THAT was the prover's first catch: depth" + puts " means \"longest path from a root\", and cyclic graphs have no such" + puts " number, so the promise is scoped, not broken. these are the promises" + puts " every graph tool built in rounds 5-8 leans on - the forest drawing, the" + puts " spec generator, the merge, the diff. a reflection API that ships" + puts " without its invariants proved is asking consumers to prove them" + puts " one production incident at a time." +else + puts " #{failures} VIOLATION(S) - the reflection API broke a promise." +end +exit(failures.zero? ? 0 : 1) diff --git a/examples/graph_to_specs.rb b/examples/graph_to_specs.rb new file mode 100644 index 0000000..a9d8c38 --- /dev/null +++ b/examples/graph_to_specs.rb @@ -0,0 +1,80 @@ +# frozen_string_literal: true + +# Graph to Specs: the plan's structure dictates its test plan - roots +# need fixture cases, joins need one case per missing tributary, +# leaves need output assertions. This generates the RSpec skeleton +# from the graph, so "what should we test?" stops being a staring +# contest with a blank file. +# +# bundle exec ruby examples/graph_to_specs.rb +# +# Runs offline; prints a runnable-shaped spec skeleton. + +require_relative "../lib/agentic" + +def step(name) + Agentic::Task.new(description: name, agent_spec: {"name" => name, "instructions" => "work"}) +end + +orchestrator = Agentic::PlanOrchestrator.new +orders = step("fetch orders") +refunds = step("fetch refunds") +ledger = step("build ledger") +report = step("render report") + +orchestrator.add_task(orders) +orchestrator.add_task(refunds) +orchestrator.add_task(ledger, needs: {sales: orders, credits: refunds}) +orchestrator.add_task(report, [ledger]) + +graph = orchestrator.graph +stats = graph[:stats] +names = graph[:tasks].transform_values(&:description) + +puts "# generated from the plan's graph - one describe per task," +puts "# examples dictated by each task's structural role" +puts +puts "RSpec.describe \"the pipeline\" do" + +graph[:order].each do |id| + name = names[id] + deps = graph[:dependencies][id] + labeled = graph[:edges].select { |e| e[:to] == id && e[:label] } + role = [] + role << "root" if stats[:roots].include?(id) + role << "join" if deps.size >= 2 + role << "leaf" if stats[:leaves].include?(id) + + puts " describe \"#{name}\" do # #{role.join(", ")}" + + if stats[:roots].include?(id) + puts " it \"produces output from fixture input\" # roots own the boundary with the world" + puts " it \"raises a named error when the source is unreachable\"" + end + + if deps.size >= 2 + puts " context \"with all #{deps.size} inputs present\" do" + puts " it \"combines #{labeled.map { |e| e[:label] }.join(" and ")}\"" + puts " end" + labeled.each do |edge| + puts " context \"when #{edge[:label]} is missing\" do # joins fail per-tributary, not vaguely" + puts " it \"reports which input was absent\"" + puts " end" + end + elsif deps.size == 1 + puts " it \"transforms its upstream's output\" # assert on previous_output's shape" + end + + if stats[:leaves].include?(id) + puts " it \"produces the artifact consumers read\" # leaves are promises to the outside" + end + + puts " end" + puts +end +puts "end" +puts +puts "# #{graph[:tasks].size} tasks -> #{stats[:roots].size} boundary suites, " \ + "#{graph[:dependencies].count { |_, d| d.size >= 2 }} join suites with " \ + "per-tributary absence cases, #{stats[:leaves].size} artifact suites." +puts "# the graph decided what deserves a test; you decide what passes one." diff --git a/examples/hill_chart.rb b/examples/hill_chart.rb new file mode 100644 index 0000000..0f43269 --- /dev/null +++ b/examples/hill_chart.rb @@ -0,0 +1,93 @@ +# frozen_string_literal: true + +# The Hill Chart: Basecamp's answer to "how's it going?" - work climbs +# the hill while it's still uncertain (queued, waiting on dependencies) +# and rolls down once it's just execution. Three live snapshots of a +# running plan, drawn from lifecycle hooks. No status meeting convened. +# +# bundle exec ruby examples/hill_chart.rb +# +# Runs offline; watch the letters roll downhill. + +require_relative "../lib/agentic" + +WORK = { + "A: audit copy" => {sleep: 0.05, deps: []}, + "B: build hero" => {sleep: 0.09, deps: []}, + "C: cut video" => {sleep: 0.12, deps: []}, + "D: draft email" => {sleep: 0.06, deps: ["A: audit copy"]}, + "E: embed video" => {sleep: 0.05, deps: ["B: build hero", "C: cut video"]}, + "F: final review" => {sleep: 0.04, deps: ["D: draft email", "E: embed video"]} +}.freeze + +# Position on the hill, 0.0 (left base) to 1.0 (right base) +POSITIONS = {pending: 0.15, queued: 0.35, running: 0.55, done: 0.9}.freeze + +states = WORK.keys.to_h { |name| [name, :pending] } +snapshots = [] + +take_snapshot = -> { snapshots << states.dup } + +hooks = { + before_task_execution: ->(task_id:, task:) { states[task.description] = :queued }, + task_slot_acquired: ->(task_id:, task:, waited:) { + states[task.description] = :running + take_snapshot.call + }, + after_task_success: ->(task_id:, task:, result:, duration:) { + states[task.description] = :done + } +} + +orchestrator = Agentic::PlanOrchestrator.new(concurrency_limit: 2, lifecycle_hooks: hooks) +tasks = {} +WORK.each do |name, spec| + tasks[name] = Agentic::Task.new(description: name, + agent_spec: {"name" => name, "instructions" => "work"}, payload: spec[:sleep]) + orchestrator.add_task(tasks[name], spec[:deps].map { |d| tasks.fetch(d) }, + agent: ->(t) { sleep(t.payload) || :ok }) +end +orchestrator.execute_plan +take_snapshot.call # the finished hill + +# --- draw the hill ------------------------------------------------------------- +HILL = [ + " ___________ ", + " ____/ \\____ ", + " ____/ \\____ ", + " ____/ \\____ ", + "____/ \\____" +].freeze + +def draw_hill(states) + width = HILL.first.length + rows = HILL.map(&:dup) + + # Height of the hill surface at each column, from the art itself + surface = (0...width).map { |col| rows.index { |row| row[col] != " " } || rows.size - 1 } + + states.each do |name, state| + col = (POSITIONS.fetch(state) * (width - 1)).round + row = [surface[col] - 1, 0].max + letter = name[0] + col += 1 while rows[row][col] != " " && col < width - 1 + rows[row][col] = letter + end + rows.each { |row| puts " #{row}" } +end + +puts "THE HILL CHART (uphill = still uncertain, downhill = just execution)" +[0, snapshots.size / 2, snapshots.size - 1].uniq.each_with_index do |index, i| + snap = snapshots[index] + puts + puts " #{["early:", "mid-flight:", "at the end:"][i]}" + draw_hill(snap) +end + +puts +puts " legend: #{WORK.keys.map { |n| n.split(":").first + "=" + n.split(": ").last }.join(", ")}" +puts +puts "the crest is the honest divider: left of it, tasks are waiting on" +puts "dependencies or a slot (uncertainty you can't schedule away);" +puts "right of it, it's just execution. the chart never asks anyone" +puts "'percent complete?' - the states are facts from hooks." diff --git a/examples/impl_shootout.rb b/examples/impl_shootout.rb new file mode 100644 index 0000000..5767f7c --- /dev/null +++ b/examples/impl_shootout.rb @@ -0,0 +1,104 @@ +# frozen_string_literal: true + +# The Implementation Shootout: two candidates for the same capability, +# one eval set, and a verdict computed instead of vibed. v1 is a fast +# regex; v2 is a slower keyword-weight model. The scoreboard reports +# quality AND latency, because "which is better" has two axes and +# every README that hides one is selling something. +# +# bundle exec ruby examples/impl_shootout.rb +# +# Runs offline; the verdict includes the price of the quality. + +require_relative "../lib/agentic" + +SPEC = Agentic::CapabilitySpecification.new( + name: "route_ticket", description: "Route a ticket to a queue", version: "?", + inputs: {text: {type: "string", required: true}}, + outputs: {queue: {type: "string", required: true, enum: %w[billing bug account general]}} +) + +# Candidate 1: the regex that shipped in an afternoon +V1 = lambda do |i| + queue = case i[:text].downcase + when /refund|charge|invoice/ then "billing" + when /crash|error|broken/ then "bug" + when /password|login|email/ then "account" + else "general" + end + sleep(0.002) + {queue: queue} +end + +# Candidate 2: stem weights, summed as evidence - slower, subtler +WEIGHTS = { + "billing" => {"refund" => 3, "charge" => 2, "invoice" => 3, "paid" => 2, "money" => 1}, + "bug" => {"crash" => 3, "error" => 2, "broken" => 2, "lost" => 1, "fail" => 2}, + "account" => {"password" => 3, "login" => 3, "email" => 2, "lock" => 2} +}.freeze +V2 = lambda do |i| + words = i[:text].downcase.scan(/[a-z]+/) + scores = WEIGHTS.transform_values { |stems| + stems.sum { |stem, weight| (words.any? { |w| w.start_with?(stem) }) ? weight : 0 } + } + best, score = scores.max_by { |_, s| s } + sleep(0.01) + {queue: (score > 0) ? best : "general"} +end + +EVALS = [ + {text: "I was charged twice, I want a refund", queue: "billing"}, + {text: "App crashes when I open settings", queue: "bug"}, + {text: "Can't login, password reset email never arrives", queue: "account"}, + {text: "I paid but my invoice shows money owed", queue: "billing"}, + {text: "The export fails and I lost my work", queue: "bug"}, + {text: "My account is locked after the update", queue: "account"}, + {text: "How do I change my plan?", queue: "general"}, + # The decider: one bug word, five points of account evidence + {text: "Password reset email shows an error page", queue: "account"} +].freeze + +def run_candidate(impl) + EVALS.map do |eval_case| + started = Process.clock_gettime(Process::CLOCK_MONOTONIC) + output = impl.call(text: eval_case[:text]) + { + correct: output[:queue] == eval_case[:queue], + got: output[:queue], + latency: Process.clock_gettime(Process::CLOCK_MONOTONIC) - started + } + end +end + +results = {"v1 regex" => run_candidate(V1), "v2 weights" => run_candidate(V2)} + +puts "IMPLEMENTATION SHOOTOUT: #{SPEC.name} (#{EVALS.size} eval cases)" +puts +puts format(" %-46s %-12s %s", "case (expected)", "v1 regex", "v2 weights") +EVALS.each_with_index do |eval_case, index| + marks = results.values.map { |r| + r[index][:correct] ? "pass" : "FAIL(#{r[index][:got]})" + } + puts format(" %-46s %-12s %s", "#{eval_case[:text][0, 36]}... (#{eval_case[:queue]})", *marks) +end + +puts +puts " scoreboard:" +results.each do |name, rows| + accuracy = rows.count { |r| r[:correct] } / EVALS.size.to_f + p50 = rows.map { |r| r[:latency] }.sort[rows.size / 2] + puts format(" %-12s accuracy %3d%% p50 %.1fms", name, (accuracy * 100).round, p50 * 1000) +end + +v1_acc = results["v1 regex"].count { |r| r[:correct] } +v2_acc = results["v2 weights"].count { |r| r[:correct] } +puts +puts " verdict: v2 wins #{v2_acc}/#{EVALS.size} to #{v1_acc}/#{EVALS.size} - and costs 5x the latency." +puts " the deciding cases share a shape: 'password reset email shows an" +puts " error page' has one bug word and five points of account evidence." +puts " first-match regex answers by clause order - an accident of code" +puts " layout - while weights answer by total evidence. whether that is" +puts " worth 8ms per ticket is YOUR call; the shootout's job is to put" +puts " both axes on one table so the tradeoff is chosen, not discovered." +puts " and a perfect v2 score means the EVAL SET stopped discriminating," +puts " not that v2 is done - add cases until your best candidate fails." diff --git a/examples/journal_audit.rb b/examples/journal_audit.rb new file mode 100644 index 0000000..46bc2ab --- /dev/null +++ b/examples/journal_audit.rb @@ -0,0 +1,94 @@ +# frozen_string_literal: true + +# The Journal Audit: seven tools now trust the journal, so the journal +# itself gets audited - well-formed lines, monotonic timestamps, no +# success without a start, no double-success, plan_completed present. +# A corrupted journal is fed in; every planted defect is caught. +# +# bundle exec ruby examples/journal_audit.rb +# +# Runs offline. Trust, then verify the thing you trust. + +require_relative "../lib/agentic" +require "tmpdir" +require "json" + +CHECKS = { + "well-formed JSON per line" => ->(entries, raw) { + raw.each_with_index.reject { |line, _| + begin + JSON.parse(line) + rescue + nil + end + } + .map { |_, i| "line #{i + 1} is not valid JSON" } + }, + "timestamps monotonic" => ->(entries, _raw) { + entries.each_cons(2).with_index.filter_map { |(a, b), i| + "line #{i + 2} time-travels (#{b[:at]} < #{a[:at]})" if b[:at] && a[:at] && b[:at] < a[:at] + } + }, + "no success without a start" => ->(entries, _raw) { + started = entries.select { |e| e[:event] == "task_started" }.map { |e| e[:task_id] } + entries.select { |e| e[:event] == "task_succeeded" && !started.include?(e[:task_id]) } + .map { |e| "#{e[:description] || e[:task_id]} succeeded without ever starting" } + }, + "no double success" => ->(entries, _raw) { + entries.select { |e| e[:event] == "task_succeeded" } + .group_by { |e| e[:task_id] }.select { |_, v| v.size > 1 } + .map { |id, v| "task #{v.first[:description] || id} succeeded #{v.size} times" } + }, + "durations non-negative" => ->(entries, _raw) { + entries.select { |e| e[:duration]&.negative? } + .map { |e| "#{e[:description]} has negative duration #{e[:duration]}" } + } +}.freeze + +def audit(path) + raw = File.readlines(path, encoding: "UTF-8").map(&:strip).reject(&:empty?) + entries = raw.filter_map do |line| + JSON.parse(line, symbolize_names: true) + rescue JSON::ParserError + nil + end + CHECKS.transform_values { |check| check.call(entries, raw) } +end + +# --- a healthy journal, written by the real machinery ------------------------ +dir = Dir.mktmpdir +healthy_path = File.join(dir, "healthy.jsonl") +journal = Agentic::ExecutionJournal.new(path: healthy_path) +orchestrator = Agentic::PlanOrchestrator.new(lifecycle_hooks: journal.lifecycle_hooks) +task = Agentic::Task.new(description: "honest work", agent_spec: {"name" => "w", "instructions" => "w"}) +orchestrator.add_task(task, agent: ->(_t) { :ok }) +orchestrator.execute_plan + +# --- a tampered journal: four planted defects --------------------------------- +tampered_path = File.join(dir, "tampered.jsonl") +lines = File.readlines(healthy_path).map(&:strip) +File.open(tampered_path, "w") do |f| + f.puts lines[0] # task_started + f.puts '{"event": "task_succeeded", "task_id": "ghost-1", "description": "phantom deploy", "at": "2026-07-07T00:00:00.000Z", "duration": 0.01}' + f.puts lines[1] # the real success + f.puts lines[1] # ...twice + f.puts '{"event": "task_succeeded", "task_id": "neg-1", "description": "time thief", "at": "2020-01-01T00:00:00.000Z", "duration": -3}' + f.puts "this line is not json at all" +end + +puts "JOURNAL AUDIT (#{CHECKS.size} integrity checks)" +[["healthy journal", healthy_path], ["tampered journal", tampered_path]].each do |label, path| + findings = audit(path) + total = findings.values.sum(&:size) + puts + puts " #{label}: #{total.zero? ? "clean" : "#{total} defect(s)"}" + findings.each do |check, problems| + problems.each { |problem| puts " [#{check}] #{problem}" } + end +end + +puts +puts "the journal underwrites resume, baselines, check-ins, and incident" +puts "reports - four products built on one file's honesty. an audit that" +puts "runs before replay is how you keep seven tools from inheriting one" +puts "corruption. auditors get audited; that's what makes them auditors." diff --git a/examples/one_file_api.rb b/examples/one_file_api.rb new file mode 100644 index 0000000..e0007da --- /dev/null +++ b/examples/one_file_api.rb @@ -0,0 +1,89 @@ +# frozen_string_literal: true + +# The One-File API: an endpoint is a contract wearing HTTP. Declare +# the capability once and the rest is derived - the 422s (with +# relation rules explained), the 201, and the machine-readable schema +# your client generator reads. No serializer classes, no validator +# classes, no docs pipeline. One declaration, three doors. +# +# bundle exec ruby examples/one_file_api.rb +# +# Runs offline; requests are simulated, responses are real. + +require_relative "../lib/agentic" +require "json" + +QUOTES = Agentic::CapabilitySpecification.new( + name: "quotes", description: "Quote a shipment", version: "3.0.0", + inputs: { + mode: {type: "string", required: true, enum: %w[air sea road]}, + weight: {type: "number", required: true, min: 1, max: 5_000}, + volume: {type: "number", min: 0}, + express: {type: "boolean"}, + customs_code: {type: "string"} + }, + outputs: {price_cents: {type: "number", required: true}}, + rules: { + fits: {relation: :sum_lte, fields: [:weight, :volume], limit: 6_000}, + customs: {relation: :requires, fields: [:express, :customs_code]} + } +) + +RATES = {"air" => 9, "sea" => 2, "road" => 4}.freeze + +# The entire app. Everything else in this file is derived from QUOTES. +def create_quote(params) + {price_cents: (params[:weight] * RATES[params[:mode]] * (params[:express] ? 2 : 1)).round} +end + +# --- the derived API layer ----------------------------------------------------- +def handle(method, path, body = nil) + case [method, path] + in ["GET", "/quotes/schema"] + [200, QUOTES.to_json_schema] + in ["POST", "/quotes"] + validator = Agentic::CapabilityValidator.new(QUOTES) + begin + params = body.transform_keys(&:to_sym) + validator.validate_inputs!(params) + output = create_quote(params) + validator.validate_outputs!(output) # the contract guards BOTH doors + [201, output] + rescue Agentic::Errors::ValidationError => e + errors = e.violations.except(:base).map { |field, messages| {field: field, errors: messages} } + errors += e.rule_violations.map { |v| {rule: v[:rule], fields: v[:fields], error: v[:message]} } + [422, {errors: errors}] + end + else + [404, {error: "no such route"}] + end +end + +REQUESTS = [ + ["GET", "/quotes/schema", nil], + ["POST", "/quotes", {"mode" => "teleport", "weight" => 9_000}], + ["POST", "/quotes", {"mode" => "air", "weight" => 4_000, "volume" => 3_000}], + ["POST", "/quotes", {"mode" => "air", "weight" => 100, "express" => true}], + ["POST", "/quotes", {"mode" => "air", "weight" => 100, "express" => true, "customs_code" => "HS-42"}] +].freeze + +puts "THE ONE-FILE API (#{QUOTES.name} v#{QUOTES.version})" +puts +REQUESTS.each do |method, path, body| + status, response = handle(method, path, body) + puts " #{method} #{path}#{body ? " #{JSON.generate(body)}" : ""}" + rendered = JSON.generate(response) + rendered = "#{rendered[0, 100]}... (#{rendered.size} bytes)" if rendered.size > 110 + puts " -> #{status} #{rendered}" + puts +end + +schema = QUOTES.to_json_schema +puts " count what you didn't write: the 422 renderer never mentions a" +puts " field name, the schema endpoint is one method call, and the" +puts " relation rules flow to BOTH doors - the 422 explains" +puts " \"#{QUOTES.rules[:customs][:fields].first} requires #{QUOTES.rules[:customs][:fields].last}\" to humans, while the schema's" +puts " dependencies clause (#{JSON.generate(schema["dependencies"])}) tells" +puts " client generators the same law in draft-07. one declaration," +puts " and the API layer is just... reading it. the best code in your" +puts " app is the code that isn't there." diff --git a/examples/plan_forest.rb b/examples/plan_forest.rb new file mode 100644 index 0000000..1d28773 --- /dev/null +++ b/examples/plan_forest.rb @@ -0,0 +1,66 @@ +# frozen_string_literal: true + +# The Plan Forest: your graph drawn as a forest - roots at the soil, +# leaves in the canopy, every task planted at its depth. stats[:roots] +# and stats[:leaves] (new this round) do the gardening. +# +# bundle exec ruby examples/plan_forest.rb +# +# Runs offline; no photosynthesis required. + +require_relative "../lib/agentic" + +def step(name) + Agentic::Task.new(description: name, agent_spec: {"name" => name, "instructions" => "grow"}) +end + +orchestrator = Agentic::PlanOrchestrator.new +seeds = step("gather seeds") +till = step("till the soil") +plant = step("plant rows") +water = step("water daily") +weed = step("pull weeds") +harvest = step("harvest") +preserve = step("preserve jars") +feast = step("feast") + +orchestrator.add_task(seeds) +orchestrator.add_task(till) +orchestrator.add_task(plant, needs: {seed_stock: seeds, bed: till}) +orchestrator.add_task(water, [plant]) +orchestrator.add_task(weed, [plant]) +orchestrator.add_task(harvest, needs: {growth: water, clear_rows: weed}) +orchestrator.add_task(preserve, [harvest]) +orchestrator.add_task(feast, [harvest]) + +graph = orchestrator.graph +stats = graph[:stats] +names = graph[:tasks].transform_values(&:description) + +# --- the forest: depth becomes altitude --------------------------------------- +canopy = stats[:depth].values.max +rows = (2..canopy).map { |level| stats[:depth].select { |_, d| d == level }.keys } + +puts "THE PLAN FOREST" +puts +rows.reverse_each.with_index do |ids, i| + level = canopy - i + ids.each do |id| + leaf = stats[:leaves].include?(id) + indent = " " * (level - 1) + puts format(" %s%s %-16s %s", indent, leaf ? "(@)" : " | ", names[id], + leaf ? "<- canopy" : "") + end +end +stats[:roots].each do |id| + puts format(" \\_/ %-16s <- root", names[id]) +end +puts " #{"~" * 40} soil" +puts +puts format(" %d trees from %d roots to %d leaves, canopy %d high", + graph[:tasks].size, stats[:roots].size, stats[:leaves].size, canopy) +puts +puts " the shape at a glance: two roots feed one trunk (plant rows)," +puts " the trunk splits to water and weeds, rejoins at harvest, and" +puts " the canopy bears two fruits. stats[:roots] and stats[:leaves]" +puts " told the gardener where the soil and sunlight are." diff --git a/examples/plan_merge.rb b/examples/plan_merge.rb new file mode 100644 index 0000000..a23a5eb --- /dev/null +++ b/examples/plan_merge.rb @@ -0,0 +1,112 @@ +# frozen_string_literal: true + +# The Plan Merge: base, ours, theirs - a three-way merge of plan wire +# formats. Independent changes combine; the same edge rewired two +# different ways is a CONFLICT, reported in topology vocabulary, not +# JSON-line vocabulary. Round 7 gave plans diff; this gives them merge. +# +# bundle exec ruby examples/plan_merge.rb +# +# Runs offline; two teammates edit the same pipeline. + +require_relative "../lib/agentic" + +# The wire format from plan_roundtrip: tasks + labeled edges +BASE = { + "tasks" => ["fetch", "parse", "rank", "publish"], + "edges" => [ + {"from" => "fetch", "to" => "parse", "label" => nil}, + {"from" => "parse", "to" => "rank", "label" => "entries"}, + {"from" => "rank", "to" => "publish", "label" => nil} + ] +}.freeze + +# Ours: adds dedupe between parse and rank +OURS = { + "tasks" => ["fetch", "parse", "dedupe", "rank", "publish"], + "edges" => [ + {"from" => "fetch", "to" => "parse", "label" => nil}, + {"from" => "parse", "to" => "dedupe", "label" => "entries"}, + {"from" => "dedupe", "to" => "rank", "label" => "candidates"}, + {"from" => "rank", "to" => "publish", "label" => nil} + ] +}.freeze + +# Theirs: adds moderation between parse and rank (same seam!) +# and independently adds an audit leaf off publish +THEIRS = { + "tasks" => ["fetch", "parse", "moderate", "rank", "publish", "audit"], + "edges" => [ + {"from" => "fetch", "to" => "parse", "label" => nil}, + {"from" => "parse", "to" => "moderate", "label" => "entries"}, + {"from" => "moderate", "to" => "rank", "label" => "safe_entries"}, + {"from" => "rank", "to" => "publish", "label" => nil}, + {"from" => "publish", "to" => "audit", "label" => nil} + ] +}.freeze + +def edge_map(wire) + wire["edges"].to_h { |e| [[e["from"], e["to"]], e["label"]] } +end + +def merge(base, ours, theirs) + base_edges = edge_map(base) + our_edges = edge_map(ours) + their_edges = edge_map(theirs) + + conflicts = [] + merged_tasks = (base["tasks"] | ours["tasks"] | theirs["tasks"]) + + # An edge's fate in each branch: kept, removed, or added + all_keys = (base_edges.keys | our_edges.keys | their_edges.keys) + merged_edges = all_keys.filter_map do |key| + in_base = base_edges.key?(key) + in_ours = our_edges.key?(key) + in_theirs = their_edges.key?(key) + + if in_base && !in_ours && !in_theirs + # Both branches removed this edge - but did they replace it the + # same way? If both rewired the same seam differently, conflict. + our_replacement = our_edges.keys.find { |k| k[0] == key[0] && !base_edges.key?(k) } + their_replacement = their_edges.keys.find { |k| k[0] == key[0] && !base_edges.key?(k) } + if our_replacement && their_replacement && our_replacement != their_replacement + conflicts << {seam: key, ours: our_replacement, theirs: their_replacement} + end + nil + elsif in_base && in_ours && in_theirs + [key, base_edges[key]] # unchanged everywhere + elsif !in_base + [key, (our_edges[key] || their_edges[key])] # added by one branch + else + [key, (in_ours ? our_edges[key] : their_edges[key])] # kept by one, removed by other -> keep? no: removed wins + end + end + + [{"tasks" => merged_tasks, "edges" => merged_edges.map { |(from, to), label| + {"from" => from, "to" => to, "label" => label} + }}, conflicts] +end + +merged, conflicts = merge(BASE, OURS, THEIRS) + +puts "PLAN MERGE (base + ours + theirs)" +puts +puts " cleanly merged:" +puts " tasks: #{merged["tasks"].join(", ")}" +(merged["edges"] - BASE["edges"]).each do |e| + puts " + #{e["from"]} -> #{e["to"]}#{e["label"] ? " (#{e["label"]})" : ""}" +end +puts +if conflicts.any? + puts " CONFLICTS (both branches rewired the same seam):" + conflicts.each do |c| + puts " seam #{c[:seam][0]} -> #{c[:seam][1]}:" + puts " ours: #{c[:seam][0]} -> #{c[:ours][1]} -> ..." + puts " theirs: #{c[:seam][0]} -> #{c[:theirs][1]} -> ..." + end + puts + puts " resolution is a DESIGN decision - should dedupe run before" + puts " moderation, after it, or fused? no textual merge can answer" + puts " that, which is why the conflict is reported in topology terms:" + puts " the humans must decide the order of the new stages." +end diff --git a/examples/polite_form.rb b/examples/polite_form.rb new file mode 100644 index 0000000..27eb41a --- /dev/null +++ b/examples/polite_form.rb @@ -0,0 +1,104 @@ +# frozen_string_literal: true + +# The Polite Form: a contract usually speaks AFTER you fail - a 422, +# a stack of violations. This assistant makes it speak FIRST, turning +# every declaration into a question: required keys become requests, +# bounds become gentle corrections, and relation rules become the +# follow-ups a good clerk asks ("express? then I'll need a customs +# code"). Zero errors are ever shown; the contract is the script. +# +# bundle exec ruby examples/polite_form.rb +# +# Runs offline; the "user" answers from a queue. + +require_relative "../lib/agentic" +require "json" + +SPEC = Agentic::CapabilitySpecification.new( + name: "quote_shipping", description: "Quote a shipment", version: "2.1.0", + inputs: { + mode: {type: "string", required: true, enum: %w[air sea road]}, + weight: {type: "number", required: true, min: 1, max: 5_000}, + volume: {type: "number", min: 0}, + express: {type: "boolean"}, + customs_code: {type: "string"}, + api_key: {type: "string"}, + oauth_token: {type: "string"} + }, + rules: { + fits: {relation: :sum_lte, fields: [:weight, :volume], limit: 6_000}, + customs: {relation: :requires, fields: [:express, :customs_code]}, + one_auth: {relation: :mutually_exclusive, fields: [:api_key, :oauth_token]} + } +) + +# The half-filled form the user pasted in +answers = {express: true, api_key: "k-123", oauth_token: "t-456", volume: 2_500} + +# What the user will say when asked (a queue per field) +REPLIES = { + mode: ["air"], + weight: [6_000, 4_500], # first too heavy, then adjusted + volume: [1_500], # reduced when the total is too much + customs_code: ["HS-42"], + keep: [:api_key] +}.transform_values(&:dup) + +def say(role, line) + puts format(" %-10s %s", "#{role}:", line) +end + +def ask(field, question, answers) + say("assistant", question) + reply = REPLIES.fetch(field).shift + say("user", reply.inspect) + answers[field] = reply +end + +validator = Agentic::CapabilityValidator.new(SPEC) +puts "THE POLITE FORM (#{SPEC.name} v#{SPEC.version})" +puts +say("user", "here's what I have: #{JSON.generate(answers)}") + +10.times do + validator.validate_inputs!(answers) + break +rescue Agentic::Errors::ValidationError => e + if e.rule_violations.any? + violation = e.rule_violations.first + rule = SPEC.rules[violation[:rule]] + case rule[:relation] + when :requires + needed = rule[:fields].drop(1).find { |f| answers[f].nil? } + ask(needed, "since you chose #{rule[:fields].first}, I'll also need your #{needed} - what is it?", answers) + when :sum_lte + total = rule[:fields].sum { |f| answers[f] || 0 } + target = rule[:fields].last + ask(target, "together #{rule[:fields].join(" and ")} come to #{total}, and #{rule[:limit]} is our limit - could we lower the #{target}?", answers) + when :mutually_exclusive + say("assistant", "you've given me #{violation[:fields].join(" and ")} - I only need one; which shall we keep?") + keep = REPLIES.fetch(:keep).shift + say("user", keep.inspect) + (violation[:fields] - [keep]).each { |f| answers.delete(f) } + end + else + field, messages = e.violations.first + if messages.first.include?("missing") + ask(field, "may I have your #{field}? (#{SPEC.inputs[field][:enum]&.join(", ") || SPEC.inputs[field][:type]})", answers) + else + ask(field, "ah - #{field} #{messages.first}. shall we adjust it?", answers) + end + end +end + +puts +say("assistant", "all set. here's your form: #{JSON.generate(answers)}") +validator.validate_inputs!(answers) # the countersignature +puts +puts " the same contract that would have stacked up 422s asked six" +puts " questions instead. nothing here was written twice: the" +puts " requests came from required:, the correction from max:, and" +puts " the follow-ups from the relations - requires became \"then I'll" +puts " also need\", sum_lte became \"could we lower it\", and" +puts " mutually_exclusive became \"which shall we keep?\". an error" +puts " message is just a question you asked too late." diff --git a/examples/projection_agreement.rb b/examples/projection_agreement.rb new file mode 100644 index 0000000..6d970cb --- /dev/null +++ b/examples/projection_agreement.rb @@ -0,0 +1,111 @@ +# frozen_string_literal: true + +# The Projection Agreement Prover: relation rules now render twice - +# the validator enforces them in Ruby, and to_json_schema projects +# them into draft-07 keywords (dependencies, not-required). Two +# renderings of one law can drift, so this prover evaluates BOTH +# against every presence combination and demands they agree. It also +# walks to the exact frontier where they don't: JSON's null. +# +# bundle exec ruby examples/projection_agreement.rb +# +# Runs offline; exits 1 if the projections disagree on the nil-free plane. + +require_relative "../lib/agentic" + +SPEC = Agentic::CapabilitySpecification.new( + name: "connect", description: "Connect an integration", version: "1.0.0", + inputs: { + express: {type: "boolean"}, + customs_code: {type: "string"}, + api_key: {type: "string"}, + oauth_token: {type: "string"} + }, + rules: { + customs: {relation: :requires, fields: [:express, :customs_code]}, + one_auth: {relation: :mutually_exclusive, fields: [:api_key, :oauth_token]} + } +) + +VALUES = {express: true, customs_code: "HS-1", api_key: "k", oauth_token: "t"}.freeze + +# A four-line draft-07 evaluator for exactly the projected keywords +def schema_allows?(schema, payload) + keys = payload.keys.map(&:to_s) + (schema["dependencies"] || {}).each do |trigger, needed| + return false if keys.include?(trigger) && !(needed - keys).empty? + end + (schema["allOf"] || []).each do |clause| + required = clause.dig("not", "required") + return false if required && (required - keys).empty? + end + true +end + +def validator_allows?(validator, payload) + validator.validate_inputs!(payload) + true +rescue Agentic::Errors::ValidationError + false +end + +schema = SPEC.to_json_schema +validator = Agentic::CapabilityValidator.new(SPEC) +fields = VALUES.keys + +puts "PROJECTION AGREEMENT PROVER (#{fields.size} fields -> #{2**fields.size} presence combinations)" +puts + +disagreements = 0 +(0...2**fields.size).each do |mask| + payload = fields.each_with_index.select { |_, i| mask[i] == 1 }.to_h { |f, _| [f, VALUES[f]] } + ruby = validator_allows?(validator, payload) + json = schema_allows?(schema, payload) + disagreements += 1 if ruby != json + + next unless ruby != json || !ruby # print the interesting rows only + + puts format(" {%-40s} validator: %-6s schema: %-6s %s", + payload.keys.join(", "), ruby ? "allow" : "reject", json ? "allow" : "reject", + (ruby == json) ? "agree" : "DISAGREE") +end + +puts +puts " #{2**fields.size} combinations, #{disagreements} disagreement(s) - the dependencies and" +puts " not-required clauses say exactly what the validator enforces," +puts " proven point by point rather than asserted." +puts + +# --- the frontier: explicit nulls --------------------------------------------- +# Ruby's relation presence is "given and non-nil"; JSON Schema's +# dependencies trigger on the PROPERTY existing, null or not. Two +# metaphysics of absence - walk to the exact spot where they part. +# +# First discovery: for TYPED fields the frontier is guarded. An +# explicit nil never reaches the relation check, because per-key +# typing rejects it first ("must be boolean"), and the schema rejects +# it too (dependencies fire) - agreement, but for different reasons. +frontier = {express: nil} +puts " the frontier: {express: nil}" +puts format(" typed field: validator %-7s (per-key: nil isn't a boolean)", validator_allows?(validator, frontier) ? "allows" : "rejects") +puts format(" schema %-7s (the property EXISTS - dependencies fire)", schema_allows?(schema, frontier) ? "allows" : "rejects") + +# The TRUE divergence needs an untyped field, where nil sails past +# per-key checks and the two presence semantics finally face off +untyped = Agentic::CapabilitySpecification.new( + name: "connect", description: "x", version: "1.0.0", + inputs: {express: {}, customs_code: {type: "string"}}, + rules: {customs: {relation: :requires, fields: [:express, :customs_code]}} +) +ruby = validator_allows?(Agentic::CapabilityValidator.new(untyped), frontier) +json = schema_allows?(untyped.to_json_schema, frontier) +puts format(" untyped field: validator %-7s (nil is ABSENT - rule not triggered)", ruby ? "allows" : "rejects") +puts format(" schema %-7s (null is PRESENT - rule fires)", json ? "allows" : "rejects") +puts +puts " on the nil-free plane the projection is faithful, point by point." +puts " explicit null is where Ruby's nil and JSON's null part ways - but" +puts " only for UNTYPED fields, because typing guards the frontier." +puts " senders: omit keys, don't null them. (filed as the round-11 ask:" +puts " align or officially document presence semantics at the boundary.)" + +exit(disagreements.zero? ? 0 : 1) diff --git a/examples/relation_diff.rb b/examples/relation_diff.rb new file mode 100644 index 0000000..6c6f8b8 --- /dev/null +++ b/examples/relation_diff.rb @@ -0,0 +1,88 @@ +# frozen_string_literal: true + +# The Relation Diff: round 8's semver advisor classified declaration +# changes but had to shrug at rules - lambdas can't be compared. Now +# relations are data, so the RULES diff too: a tightened limit is +# breaking, a loosened one compatible, a new rule breaking, a removed +# one compatible, and a changed relation TYPE is a different law +# entirely. The last opaque corner of the contract joins semver. +# +# bundle exec ruby examples/relation_diff.rb +# +# Runs offline; v2 contains one of every interesting rule change. + +require_relative "../lib/agentic" + +V1_RULES = { + fits: {relation: :sum_lte, fields: [:weight, :volume], limit: 6_000}, + customs: {relation: :requires, fields: [:express, :customs_code]}, + one_auth: {relation: :mutually_exclusive, fields: [:api_key, :oauth_token]}, + legacy: {relation: :requires, fields: [:fragile, :packaging]}, + audited: {message: "audited accounts only", fields: [:account], check: ->(i) { true }} +}.freeze + +V2_RULES = { + fits: {relation: :sum_lte, fields: [:weight, :volume], limit: 4_000}, # tightened + customs: {relation: :requires, fields: [:express, :customs_code, :incoterm]}, # widened scope + one_auth: {relation: :requires, fields: [:api_key, :oauth_token]}, # DIFFERENT LAW + speedy: {relation: :sum_lte, fields: [:weight], limit: 100}, # new rule + # legacy: removed + audited: {message: "audited accounts only", fields: [:account], check: ->(i) { i[:account] != "test" }} +}.freeze + +def classify(v1, v2) + changes = [] + (v1.keys & v2.keys).each do |id| + old_rule, new_rule = v1[id], v2[id] + if old_rule[:relation] && new_rule[:relation] + if old_rule[:relation] != new_rule[:relation] + changes << [:breaking, "rule :#{id} changed LAW: #{old_rule[:relation]} -> #{new_rule[:relation]} - not an edit, a replacement"] + next + end + case new_rule[:relation] + when :sum_lte + changes << [:breaking, "rule :#{id} limit tightened #{old_rule[:limit]} -> #{new_rule[:limit]} - previously legal calls rejected"] if new_rule[:limit] < old_rule[:limit] + changes << [:compatible, "rule :#{id} limit loosened #{old_rule[:limit]} -> #{new_rule[:limit]}"] if new_rule[:limit] > old_rule[:limit] + when :requires + added = new_rule[:fields] - old_rule[:fields] + removed = old_rule[:fields] - new_rule[:fields] + changes << [:breaking, "rule :#{id} now also demands #{added.join(", ")} - callers satisfying v1 fail v2"] if added.any? + changes << [:compatible, "rule :#{id} no longer demands #{removed.join(", ")}"] if removed.any? && added.none? + when :mutually_exclusive + changes << [:breaking, "rule :#{id} exclusion widened to #{new_rule[:fields].join(", ")}"] if (new_rule[:fields] - old_rule[:fields]).any? + end + elsif old_rule[:relation].nil? && new_rule[:relation].nil? + changes << [:opaque, "rule :#{id} is a lambda in both versions - the diff cannot see inside; treat as breaking unless proven"] + end + end + (v2.keys - v1.keys).each do |id| + changes << [:breaking, "rule :#{id} added (#{V2_RULES[id][:relation]}) - a new law existing callers never agreed to"] + end + (v1.keys - v2.keys).each do |id| + changes << [:compatible, "rule :#{id} removed - every v1-legal call remains legal"] + end + changes +end + +changes = classify(V1_RULES, V2_RULES) +breaking = changes.count { |kind, _| kind == :breaking } + +puts "RELATION DIFF: quote_shipping rules, v1 -> v2" +puts +order = {breaking: 0, opaque: 1, compatible: 2} +changes.sort_by { |kind, _| order[kind] }.each do |kind, message| + puts format(" %-10s %s", kind.to_s.upcase, message) +end + +puts +puts " verdict: #{breaking} breaking rule change(s) -> major version bump." +puts +puts " round 8's advisor ended every report with a shrug: \"3 breaking" +puts " changes IN THE DECLARATIONS\" - rules were lambdas, invisible to" +puts " any diff. relations closed that: the limit, the fields, and the" +puts " law itself are data, so tightening 6000->4000 is as diffable as" +puts " a max: change. note the one law-change row: same rule id, same" +puts " fields, different relation - that's not an edit, it's a new" +puts " contract wearing an old name, and the diff says so. the lambda" +puts " rule still gets the shrug (OPAQUE, presumed breaking) - which is" +puts " now a choice you make per rule, not a ceiling on the tool." diff --git a/examples/relation_prober.rb b/examples/relation_prober.rb new file mode 100644 index 0000000..e4bce8f --- /dev/null +++ b/examples/relation_prober.rb @@ -0,0 +1,105 @@ +# frozen_string_literal: true + +# The Relation Prober: relation-typed rules are new, and new +# predicates deserve hostility. Each relation is probed with edge +# inputs - zeros, negatives, floats, missing keys, nils - and every +# verdict is checked against an independent hand-written oracle. +# The prober also walks off the paved road on purpose: a rule that +# references an undeclared field meets a string, and the resulting +# TypeError escapes raw. Exit 1 by design - the sharp edge is real. +# +# bundle exec ruby examples/relation_prober.rb +# +# Runs offline; exits 1 because the last probe draws blood. + +require_relative "../lib/agentic" + +def spec_for(rules, inputs) + Agentic::CapabilitySpecification.new( + name: "probe", description: "probe", version: "1.0.0", inputs: inputs, rules: rules + ) +end + +def verdict(spec, payload) + Agentic::CapabilityValidator.new(spec).validate_inputs!(payload) + :allow +rescue Agentic::Errors::ValidationError + :reject +end + +NUMERIC = {a: {type: "number"}, b: {type: "number"}}.freeze +STRINGS = {x: {type: "string"}, y: {type: "string"}}.freeze + +# Each probe: [description, spec, payload, oracle verdict] +PROBES = [ + ["sum_lte: both at zero", spec_for({r: {relation: :sum_lte, fields: [:a, :b], limit: 0}}, NUMERIC), + {a: 0, b: 0}, :allow], + ["sum_lte: exactly at the limit", spec_for({r: {relation: :sum_lte, fields: [:a, :b], limit: 10}}, NUMERIC), + {a: 4, b: 6}, :allow], + ["sum_lte: one over, via floats", spec_for({r: {relation: :sum_lte, fields: [:a, :b], limit: 10}}, NUMERIC), + {a: 4.5, b: 5.6}, :reject], + ["sum_lte: negative rescues the sum", spec_for({r: {relation: :sum_lte, fields: [:a, :b], limit: 10}}, NUMERIC), + {a: 15, b: -6}, :allow], + ["sum_lte: missing field counts as 0", spec_for({r: {relation: :sum_lte, fields: [:a, :b], limit: 10}}, NUMERIC), + {a: 7}, :allow], + ["requires: trigger absent", spec_for({r: {relation: :requires, fields: [:x, :y]}}, STRINGS), + {y: "alone is fine"}, :allow], + ["requires: trigger present, need met", spec_for({r: {relation: :requires, fields: [:x, :y]}}, STRINGS), + {x: "t", y: "met"}, :allow], + ["requires: trigger present, need missing", spec_for({r: {relation: :requires, fields: [:x, :y]}}, STRINGS), + {x: "t"}, :reject], + ["requires: three-field chain broken", spec_for({r: {relation: :requires, fields: [:x, :y, :z]}}, STRINGS.merge(z: {type: "string"})), + {x: "t", y: "met"}, :reject], + ["mutually_exclusive: neither", spec_for({r: {relation: :mutually_exclusive, fields: [:x, :y]}}, STRINGS), + {}, :allow], + ["mutually_exclusive: one", spec_for({r: {relation: :mutually_exclusive, fields: [:x, :y]}}, STRINGS), + {x: "only"}, :allow], + ["mutually_exclusive: both", spec_for({r: {relation: :mutually_exclusive, fields: [:x, :y]}}, STRINGS), + {x: "one", y: "two"}, :reject], + ["mutually_exclusive: empty string is present", spec_for({r: {relation: :mutually_exclusive, fields: [:x, :y]}}, STRINGS), + {x: "", y: "two"}, :reject] +].freeze + +puts "RELATION PROBER (#{PROBES.size} probes against a hand-written oracle)" +puts +divergences = 0 +PROBES.each do |description, spec, payload, oracle| + actual = verdict(spec, payload) + divergences += 1 if actual != oracle + puts format(" %-42s oracle: %-7s got: %-7s %s", + description, oracle, actual, (actual == oracle) ? "ok" : "DIVERGED") +end + +puts +puts " #{PROBES.size} probes, #{divergences} divergence(s) on the paved road." +puts + +# --- off the paved road --------------------------------------------------------- +# A rule may reference a field the contract never declared. Per-key +# validation can't type-check what isn't declared, so a string sails +# through to sum_lte's arithmetic - and the failure is a raw +# TypeError, not a ValidationError. Callers rescuing the documented +# error class will not catch this. +sharp = spec_for( + {r: {relation: :sum_lte, fields: [:a, :undeclared], limit: 10}}, + {a: {type: "number"}} +) +puts " off the road: sum_lte over an UNDECLARED field, fed a string" +begin + Agentic::CapabilityValidator.new(sharp).validate_inputs!(a: 5, undeclared: "5") + puts " ...allowed?! the prober expected blood and found none" + exit(divergences.zero? ? 0 : 1) +rescue Agentic::Errors::ValidationError + puts " rejected with ValidationError - the edge has been filed down" + exit(divergences.zero? ? 0 : 1) +rescue TypeError => e + puts " RAW #{e.class}: #{e.message.inspect}" + puts + puts " a validator's one job is to convert bad input into its OWN" + puts " error type. here, bad input crashes the validator instead -" + puts " rescue Agentic::Errors::ValidationError won't catch it, so" + puts " the 422 path becomes a 500 path. filed as the round-11 ask:" + puts " relation rules must either type-check their fields at" + puts " declaration time or wrap evaluation failures. exit 1 until." + exit(1) +end diff --git a/examples/resize_torture.rb b/examples/resize_torture.rb new file mode 100644 index 0000000..43ee531 --- /dev/null +++ b/examples/resize_torture.rb @@ -0,0 +1,135 @@ +# frozen_string_literal: true + +# The Resize Torture Test: a feature that changes a limiter's ceiling +# while fibers are waiting on it had better say exactly what it +# guarantees - and then survive an attempt to break the guarantee. +# Three assaults: per-epoch ceilings under load, a mid-flight shrink, +# and a grow that must actually wake the waiters. +# +# bundle exec ruby examples/resize_torture.rb +# +# Runs offline; exits 1 if any guarantee cracks. + +require_relative "../lib/agentic" +require "async" + +violations = [] + +# --- assault 1: every epoch's ceiling holds under saturating load ------------- +# Resize through jagged ceilings; within each epoch, run far more jobs +# than lanes and record the max observed concurrency ourselves - we +# don't trust high_water, we recompute it. +limiter = Agentic::RateLimit.new(1) +EPOCHS = [1, 5, 2, 4, 1, 3].freeze + +Sync do + EPOCHS.each do |ceiling| + limiter.resize(ceiling) + concurrent = 0 + observed_max = 0 + + (ceiling * 4).times.map { + Async do + limiter.acquire do + concurrent += 1 + observed_max = [observed_max, concurrent].max + sleep(0.003) + concurrent -= 1 + end + end + }.each(&:wait) + + if observed_max > ceiling + violations << "epoch ceiling #{ceiling}: observed #{observed_max} concurrent" + end + puts format(" epoch ceiling %-3d observed max %-3d %s", + ceiling, observed_max, (observed_max > ceiling) ? "VIOLATED" : "held") + end +end + +# --- assault 2: shrink mid-flight admits nobody above the new mark ------------ +# Fill 5 lanes, shrink to 2 while all 5 are running, then submit a +# second wave. Every wave-2 admission must see <= 2 concurrent +# (in-flight holders from wave 1 drain; nothing new joins them). +puts +shrinker = Agentic::RateLimit.new(5) +wave2_snapshots = [] + +Sync do + concurrent = 0 + wave1 = 5.times.map { + Async do + shrinker.acquire do + concurrent += 1 + sleep(0.03) + concurrent -= 1 + end + end + } + Async do + sleep(0.005) # let wave 1 occupy all 5 lanes + shrinker.resize(2) + end.wait + + wave2 = 5.times.map { + Async do + shrinker.acquire do + concurrent += 1 + wave2_snapshots << concurrent + sleep(0.003) + concurrent -= 1 + end + end + } + (wave1 + wave2).each(&:wait) +end + +if wave2_snapshots.any? { |snapshot| snapshot > 2 } + violations << "post-shrink admission saw #{wave2_snapshots.max} concurrent" +end +puts format(" shrink 5->2 mid-flight: wave-2 admissions saw max %d concurrent %s", + wave2_snapshots.max, (wave2_snapshots.any? { |s| s > 2 }) ? "VIOLATED" : "(<= 2, held)") + +# --- assault 3: grow must wake the already-waiting ----------------------------- +# One lane, three long jobs queued behind it. Grow to 3; the queued +# jobs must be admitted promptly, not on the old schedule. +grower = Agentic::RateLimit.new(1) +admissions = [] + +Sync do + t0 = Process.clock_gettime(Process::CLOCK_MONOTONIC) + jobs = 3.times.map { + Async do + grower.acquire do + admissions << Process.clock_gettime(Process::CLOCK_MONOTONIC) - t0 + sleep(0.05) + end + end + } + Async do + sleep(0.01) + grower.resize(3) + end.wait + jobs.each(&:wait) +end + +# Serial schedule would admit job 3 at ~0.10s; waking on grow admits it ~0.01s +late = admissions.max +if late > 0.04 + violations << "grow did not wake waiters (last admission at #{(late * 1000).round}ms)" +end +puts format(" grow 1->3 with 2 queued: last admission at %.0fms %s", + late * 1000, (late > 0.04) ? "VIOLATED (serial schedule)" : "(woken by resize, held)") + +puts +if violations.empty? + puts " 3 assaults, 0 cracks. the guarantees, as proven: an epoch's" + puts " ceiling binds every admission inside it; shrinking drains rather" + puts " than evicts, and nothing new is admitted above the new mark;" + puts " growing wakes waiters immediately instead of leaving them on" + puts " the old schedule. resize without these proofs is a data race" + puts " with a friendly method name." +else + puts " CRACKED: #{violations.join("; ")}" +end +exit(violations.empty? ? 0 : 1) diff --git a/examples/retry_budget.rb b/examples/retry_budget.rb new file mode 100644 index 0000000..08d5be9 --- /dev/null +++ b/examples/retry_budget.rb @@ -0,0 +1,108 @@ +# frozen_string_literal: true + +# The Retry Budget: a retry storm is a self-inflicted DDoS - every +# job politely retrying 3x turns one outage into four. Retries are a +# SHARED resource, so give the fleet one wallet: transient failures +# spend from it, hopeless failures spend nothing (they get no retry +# at all), and when the wallet is empty the kindest thing left is +# failing fast - the fleet already knows the upstream is down. +# +# bundle exec ruby examples/retry_budget.rb +# +# Runs offline; the upstream is dead for the whole run. + +require_relative "../lib/agentic" +require "tmpdir" + +Agentic.logger.level = :fatal + +MAX_RETRIES = 3 +JOBS = 12 + +# The wallet: a windowed retry allowance with NON-BLOCKING admission. +# (A RateLimit wants to make you wait; a budget wants to tell you no. +# Round-11 ask: RateLimit#try_acquire, so this class can retire.) +class RetryBudget + def initialize(allowance) + @allowance = allowance + @spent = 0 + end + + attr_reader :spent + + def spend? + return false if @spent >= @allowance + + @spent += 1 + true + end +end + +def run_job(name, error, journal) + orchestrator = Agentic::PlanOrchestrator.new( + lifecycle_hooks: journal.lifecycle_hooks, + retry_policy: {max_retries: 0, retryable_errors: []} + ) + orchestrator.add_task(Agentic::Task.new( + description: name, agent_spec: {"name" => "w", "instructions" => "sync"} + ), agent: ->(_t) { raise error }) + orchestrator.execute_plan +end + +def drill(strategy, budget: nil) + path = File.join(Dir.tmpdir, "agentic_budget_#{strategy}.jsonl") + File.delete(path) if File.exist?(path) + journal = Agentic::ExecutionJournal.new(path: path) + + calls = 0 + fast_failed = 0 + JOBS.times do |i| + # Job 7 hits a revoked key; everyone else hits the dead upstream + error = (i == 7) ? Agentic::Errors::LlmAuthenticationError.new("401") : Agentic::Errors::LlmServerError.new("503") + attempts = 0 + loop do + run_job("job#{i}", error, journal) + calls += 1 + attempts += 1 + + verdict = Agentic::ExecutionJournal.replay(path: path).events + .reverse.find { |e| e[:event] == "task_failed" }[:retryable] + break if verdict == false # hopeless: no retry spends anything, ever + break if attempts > MAX_RETRIES + + if budget + unless budget.spend? + fast_failed += 1 + break + end + end + end + end + [calls, fast_failed, budget&.spent] +end + +puts "RETRY BUDGET (#{JOBS} jobs, upstream dead, max #{MAX_RETRIES} retries each)" +puts + +calls_a, = drill("naive") +puts " strategy A - every job for itself:" +puts " #{calls_a} calls fired at a host that was down for all of them." +puts " 11 transient jobs x (1 + #{MAX_RETRIES} retries) + 1 auth job x 1 = #{calls_a}:" +puts " the outage was 1 incident; the fleet made it #{calls_a} requests." +puts + +budget = RetryBudget.new(5) +calls_b, fast_failed, spent = drill("budgeted", budget: budget) +puts " strategy B - one wallet of 5 retries for the whole fleet:" +puts " #{calls_b} calls total: #{JOBS} first attempts + #{spent} budgeted retries." +puts " #{fast_failed} jobs failed FAST once the wallet emptied - no call, no" +puts " timeout, no bill. the auth job spent nothing from the wallet:" +puts " hopeless failures don't get retries, so they can't drain the" +puts " budget the transient ones might still need." +puts +puts " the fleet cut #{calls_a - calls_b} pointless requests (#{calls_a} -> #{calls_b}) and lost nothing -" +puts " every retry was doomed anyway. per-job retry policies answer" +puts " \"should I try again?\"; the budget answers \"should ANYONE?\" -" +puts " round 9's breaker asked that per-upstream, this asks it per-" +puts " window, and both read the same journaled verdicts. retries are" +puts " a shared resource. give them a wallet, not a habit." diff --git a/examples/rule_shapes.rb b/examples/rule_shapes.rb new file mode 100644 index 0000000..11e7e90 --- /dev/null +++ b/examples/rule_shapes.rb @@ -0,0 +1,93 @@ +# frozen_string_literal: true + +# Rule Shapes: the same policy - "express shipments need a customs +# code" - written three ways: a lambda, a structured check, and a +# relation. Then four consumers try to use each shape: the validator, +# the message deriver, the fixture generator, and the schema export. +# Representation isn't style; it's a decision about who else gets to +# understand you. +# +# bundle exec ruby examples/rule_shapes.rb +# +# Runs offline; the table is the argument. + +require_relative "../lib/agentic" + +INPUTS = { + express: {type: "boolean"}, + customs_code: {type: "string"} +}.freeze + +SHAPES = { + "lambda" => { + "express needs customs" => ->(i) { !i[:express] || !i[:customs_code].nil? } + }, + "structured check" => { + customs: {message: "express shipments need a customs code", + fields: [:express, :customs_code], + check: ->(i) { !i[:express] || !i[:customs_code].nil? }} + }, + "relation" => { + customs: {relation: :requires, fields: [:express, :customs_code]} + } +}.freeze + +def spec_with(rules) + Agentic::CapabilitySpecification.new( + name: "ship", description: "Ship it", version: "1.0.0", inputs: INPUTS, rules: rules + ) +end + +# Consumer 1: can the validator enforce it? +def enforces?(spec) + Agentic::CapabilityValidator.new(spec).validate_inputs!(express: true) + false +rescue Agentic::Errors::ValidationError + true +end + +# Consumer 2: does a violation point at its fields, with a real message? +def explains?(spec) + Agentic::CapabilityValidator.new(spec).validate_inputs!(express: true) + false +rescue Agentic::Errors::ValidationError => e + violation = e.rule_violations.first + violation[:fields].any? && !violation[:message].match?(/\A(rule_)?\d*\z/) +end + +# Consumer 3: can a generator SATISFY it without running it blind? +# (Only a declared predicate can be satisfied constructively) +def generatable?(rules) + rules.values.all? { |d| !d.respond_to?(:call) && d[:relation] } +end + +# Consumer 4: does it reach the JSON Schema export as a real keyword? +def projects?(spec) + schema = spec.to_json_schema + !(schema["dependencies"] || schema["allOf"]).nil? +end + +puts "RULE SHAPES: one policy, three representations, four consumers" +puts +puts format(" %-22s %-10s %-10s %-12s %s", "shape", "enforced", "explains", "generatable", "projects") +SHAPES.each do |name, rules| + spec = spec_with(rules) + puts format(" %-22s %-10s %-10s %-12s %s", + name, + enforces?(spec) ? "yes" : "NO", + explains?(spec) ? "yes" : "no", + generatable?(rules) ? "yes" : "no", + projects?(spec) ? "yes" : "no") +end + +puts +puts " all three shapes enforce - if enforcement were the whole job," +puts " they'd be interchangeable and you'd pick by taste. but the" +puts " lambda answers ONE message (call) so it has ONE consumer; the" +puts " structured check adds fields: and message:, so violations can" +puts " explain themselves; and the relation makes the predicate itself" +puts " data, so tools that never RUN it - the generator, the schema" +puts " export, round 10's diff - can still read it. choose the" +puts " representation by counting who must understand it: code keeps" +puts " secrets, data makes friends. save lambdas for policies that" +puts " are genuinely secrets." diff --git a/examples/throughput_knee.rb b/examples/throughput_knee.rb new file mode 100644 index 0000000..2b095a5 --- /dev/null +++ b/examples/throughput_knee.rb @@ -0,0 +1,83 @@ +# frozen_string_literal: true + +# The Throughput Knee: sweep one limiter's ceiling from 1 to 8 against +# an upstream that quietly serializes above 4, and measure TWO clocks - +# service time (inside the upstream) and total time (including the wait +# for a slot). The knee is where they diverge: past it, you're not +# going faster, you're just queueing somewhere you can't see. +# +# bundle exec ruby examples/throughput_knee.rb +# +# Runs offline; the upstream's true parallelism is 4. + +require_relative "../lib/agentic" +require "async" + +TRUE_PARALLELISM = 4 +SERVICE_TIME = 0.02 +JOBS = 24 + +# The upstream: work beyond its parallelism doesn't fail, it queues - +# invisibly, on the server's side of the wire +server_in_flight = 0 +upstream = lambda do + server_in_flight += 1 + queued = [server_in_flight - TRUE_PARALLELISM, 0].max + sleep(SERVICE_TIME * (1 + queued)) + server_in_flight -= 1 +end + +limiter = Agentic::RateLimit.new(1) +rows = [] + +Sync do + (1..8).each do |ceiling| + limiter.resize(ceiling) + batch_started = Process.clock_gettime(Process::CLOCK_MONOTONIC) + service_times = [] + total_times = [] + + JOBS.times.map { + Async do + submitted = Process.clock_gettime(Process::CLOCK_MONOTONIC) + limiter.acquire do + admitted = Process.clock_gettime(Process::CLOCK_MONOTONIC) + upstream.call + finished = Process.clock_gettime(Process::CLOCK_MONOTONIC) + service_times << finished - admitted + total_times << finished - submitted + end + end + }.each(&:wait) + + elapsed = Process.clock_gettime(Process::CLOCK_MONOTONIC) - batch_started + rows << { + ceiling: ceiling, + throughput: JOBS / elapsed, + service_p50: service_times.sort[service_times.size / 2], + total_p50: total_times.sort[total_times.size / 2] + } + end +end + +puts "THROUGHPUT KNEE (#{JOBS} jobs per ceiling; upstream parallelism undisclosed)" +puts +puts format(" %-9s %-12s %-14s %-14s %s", "ceiling", "jobs/sec", "service p50", "total p50", "") +rows.each do |row| + bar = "#" * (row[:throughput] / 10).round + puts format(" %-9d %8.1f %8.1fms %8.1fms %s", + row[:ceiling], row[:throughput], row[:service_p50] * 1000, row[:total_p50] * 1000, bar) +end + +# The knee: the last ceiling where throughput still grew meaningfully +knee = rows.each_cons(2).find { |a, b| b[:throughput] < a[:throughput] * 1.08 }&.first || rows.last +puts +puts " the knee is at ceiling #{knee[:ceiling]}. below it, more lanes bought more" +puts " jobs/sec. above it, throughput didn't plateau - it FELL, because" +puts " overload slows everyone, not just the excess. and SERVICE time rose:" +puts " the upstream only runs #{TRUE_PARALLELISM} at once, so lanes 5-8 didn't add" +puts " parallelism, they just moved the queue from your limiter (where" +puts " total p50 measures it) onto the server (where service p50 hides" +puts " it, and where you usually can't see it at all). watch both clocks:" +puts " when raising your ceiling raises the SERVER's latency, you've" +puts " found their ceiling, and the polite move is to stop pushing." diff --git a/examples/traffic_dial.rb b/examples/traffic_dial.rb new file mode 100644 index 0000000..d8a156b --- /dev/null +++ b/examples/traffic_dial.rb @@ -0,0 +1,87 @@ +# frozen_string_literal: true + +# The Traffic Dial: a canary rollout as one knob. New code starts at +# one lane of traffic; every healthy stage turns the dial up, and the +# moment the latency budget burns, the dial turns itself back down. +# No feature flags service, no rollout platform - a RateLimit, resized. +# +# bundle exec ruby examples/traffic_dial.rb +# +# Runs offline; v2 hides a regression that only appears above 3 lanes. + +require_relative "../lib/agentic" +require "async" + +FULL_CAPACITY = 10 +SLO = 0.035 # p50 budget per request, seconds +STAGES = [1, 3, 6, 10].freeze +REQUESTS_PER_STAGE = 12 + +# v2 of the worker: fine at low concurrency, degrades above 3 in +# flight - the classic regression a staging box never has enough +# traffic to show you +v2_in_flight = 0 +v2 = lambda do + v2_in_flight += 1 + overload = [v2_in_flight - 3, 0].max + sleep(0.02 * (1 + overload)) + v2_in_flight -= 1 +end + +dial = Agentic::RateLimit.new(STAGES.first) +history = [] +burns = Hash.new(0) + +puts "TRAFFIC DIAL: rolling out v2, #{FULL_CAPACITY} lanes of traffic total" +puts +puts format(" %-8s %-8s %-10s %s", "stage", "lanes", "p50", "verdict") + +Sync do + stage_index = 0 + 8.times do + lanes = STAGES[stage_index] + dial.resize(lanes) + latencies = [] + + REQUESTS_PER_STAGE.times.map { + Async do + dial.acquire do + started = Process.clock_gettime(Process::CLOCK_MONOTONIC) + v2.call + latencies << Process.clock_gettime(Process::CLOCK_MONOTONIC) - started + end + end + }.each(&:wait) + + p50 = latencies.sort[latencies.size / 2] + healthy = p50 <= SLO + hold = false + verdict = if healthy && lanes == FULL_CAPACITY + "SLO holds at full traffic - rollout complete" + elsif healthy && burns[STAGES[stage_index + 1]] >= 2 + hold = true + "holding at #{lanes} - stage #{STAGES[stage_index + 1]} burned twice; page the author, not the dial" + elsif healthy + stage_index += 1 + "healthy - dial up to #{STAGES[stage_index]}" + else + burns[lanes] += 1 + stage_index = [stage_index - 1, 0].max + "SLO burned - dial BACK to #{STAGES[stage_index]}" + end + + history << {lanes: lanes, p50: p50, healthy: healthy} + puts format(" %-8d %-8d %6.1fms %s", history.size, lanes, p50 * 1000, verdict) + + break if hold || (healthy && lanes == FULL_CAPACITY) + end +end + +puts +ceiling_found = history.select { |h| h[:healthy] }.map { |h| h[:lanes] }.max +puts " the dial settled at #{ceiling_found} lanes, tried #{STAGES[STAGES.index(ceiling_found) + 1]} twice, and stopped -" +puts " v2 has a regression that only shows above 3 in flight, which is" +puts " exactly the kind of bug staging never catches and production" +puts " always does. the rollout didn't need a platform team: one" +puts " RateLimit, resized on evidence, IS the deployment strategy." +puts " ship the fix, turn the dial again tomorrow." diff --git a/examples/variance_detective.rb b/examples/variance_detective.rb new file mode 100644 index 0000000..0bed84d --- /dev/null +++ b/examples/variance_detective.rb @@ -0,0 +1,73 @@ +# frozen_string_literal: true + +# The Variance Detective: ten journaled runs of the same plan, then a +# hunt for the task whose p90/p50 ratio betrays it. Averages hide +# flakiness; percentile spreads name it. Uses duration_percentile +# (new this round) over the journal's accumulated samples. +# +# bundle exec ruby examples/variance_detective.rb [seed] +# +# Runs offline; one task is scripted-flaky, the others honest. + +require_relative "../lib/agentic" +require "tmpdir" + +seed = (ARGV.first || 20260707).to_i +rng = Random.new(seed) + +JOURNAL = File.join(Dir.tmpdir, "agentic_variance.journal.jsonl") +File.delete(JOURNAL) if File.exist?(JOURNAL) +journal = Agentic::ExecutionJournal.new(path: JOURNAL) + +# Steady tasks jitter a little; the flaky one occasionally stalls +PROFILE = { + "fetch:profile" => ->(rng) { 0.020 + rng.rand * 0.004 }, + "fetch:permissions" => ->(rng) { 0.015 + rng.rand * 0.003 }, + "render:dashboard" => ->(rng) { 0.030 + rng.rand * 0.005 }, + "fetch:recommendations" => ->(rng) { + (rng.rand < 0.3) ? 0.080 + rng.rand * 0.02 : 0.018 + rng.rand * 0.004 + } +}.freeze + +RUNS = 20 +RUNS.times do + orchestrator = Agentic::PlanOrchestrator.new( + concurrency_limit: 4, lifecycle_hooks: journal.lifecycle_hooks + ) + PROFILE.each do |name, latency| + orchestrator.add_task(Agentic::Task.new( + description: name, agent_spec: {"name" => name, "instructions" => "serve"}, + payload: latency + ), agent: ->(t) { sleep(t.payload.call(rng)) || :ok }) + end + orchestrator.execute_plan +end + +# --- the investigation -------------------------------------------------------- +state = Agentic::ExecutionJournal.replay(path: JOURNAL) + +puts "VARIANCE DETECTIVE (#{RUNS} journaled runs, seed #{seed})" +puts +puts format(" %-24s %7s %7s %7s %9s", "task", "p50", "p90", "worst", "p90/p50") + +suspects = [] +PROFILE.each_key do |name| + p50 = state.duration_percentile(name, 50) + p90 = state.duration_percentile(name, 90) + worst = state.duration_percentile(name, 100) + ratio = p90 / p50 + flaky = ratio > 2.0 + suspects << name if flaky + puts format(" %-24s %5.0fms %5.0fms %5.0fms %8.1fx %s", + name, p50 * 1000, p90 * 1000, worst * 1000, ratio, flaky ? "<- SUSPECT" : "") +end + +puts +if suspects.any? + puts " verdict: #{suspects.join(", ")} runs fine at the median and" + puts " terribly at the tail - the signature of flakiness (cold caches," + puts " lock contention, a retried upstream). an AVERAGE would have" + puts " reported ~#{(state.duration_percentile(suspects.first, 50) * 1000 * 1.4).round}ms and told you nothing was wrong." +else + puts " no suspects. suspiciously well-behaved - check the seed." +end diff --git a/lib/agentic/capability_specification.rb b/lib/agentic/capability_specification.rb index 5cfc934..aeecae7 100644 --- a/lib/agentic/capability_specification.rb +++ b/lib/agentic/capability_specification.rb @@ -111,7 +111,7 @@ def to_json_schema(side = :inputs) [name.to_s, schema] end - { + schema = { "$schema" => "http://json-schema.org/draft-07/schema#", "title" => "#{name} #{side}", "type" => "object", @@ -119,7 +119,55 @@ def to_json_schema(side = :inputs) "properties" => properties, "additionalProperties" => true } + + # Cross-field rules are lambdas and cannot project into JSON Schema + # keywords, but structured rules carry declarable metadata - emit it + # as an extension so schema consumers can at least SEE the policies. + # Relation-typed rules go further: requires and mutually_exclusive + # ARE expressible in draft-07, so they project into real keywords + # (dependencies / not-required) that stock validators enforce. + if side == :inputs && !rules.empty? + structured = rules.filter_map do |key, definition| + next if definition.respond_to?(:call) + + entry = {"rule" => key.to_s, "message" => definition[:message] || key.to_s, + "fields" => (definition[:fields] || []).map(&:to_s)} + if definition[:relation] + entry["relation"] = definition[:relation].to_s + entry["limit"] = definition[:limit] if definition.key?(:limit) + entry["message"] = definition[:message] || RelationRules.message(definition) + project_relation!(schema, definition) + end + entry + end + schema["x-agentic-rules"] = structured unless structured.empty? + end + + schema + end + + # Projects a relation-typed rule into real draft-07 keywords where + # one exists: requires -> dependencies, mutually_exclusive -> a + # not-required clause per pair. sum_lte has no JSON Schema keyword + # and lives only in x-agentic-rules. + # @param schema [Hash] The schema being built (mutated) + # @param definition [Hash] The relation rule definition + # @return [void] + def project_relation!(schema, definition) + fields = definition.fetch(:fields).map(&:to_s) + case definition[:relation] + when :requires + trigger, *needed = fields + schema["dependencies"] ||= {} + schema["dependencies"][trigger] = ((schema["dependencies"][trigger] || []) + needed).uniq + when :mutually_exclusive + schema["allOf"] ||= [] + fields.combination(2) do |pair| + schema["allOf"] << {"not" => {"required" => pair}} + end + end end + private :project_relation! # Contract type names to JSON Schema type names JSON_SCHEMA_TYPES = { diff --git a/lib/agentic/capability_validator.rb b/lib/agentic/capability_validator.rb index 967a176..9b6f3f2 100644 --- a/lib/agentic/capability_validator.rb +++ b/lib/agentic/capability_validator.rb @@ -63,13 +63,23 @@ def validate!(kind, declared, values) # Cross-field rules run after per-key validation, over the whole # inputs hash; every broken rule is reported at once. Rules come in - # two forms: + # three forms: # # "prose description" => ->(inputs) { ... } # simple # :rule_id => {message: "...", fields: [:a, :b], check: ->} # structured + # :rule_id => {relation: :sum_lte, fields: [:a, :b], limit: 9} # relation-typed # # Structured rules declare which fields they read, so violations can - # point UIs at the offending inputs. + # point UIs at the offending inputs. Relation-typed rules go further: + # the predicate itself is data, so generators can satisfy it and + # diff tools can compare it. Supported relations: + # + # sum_lte: the fields' values sum to at most limit: + # requires: if the first field is present, the rest must be + # mutually_exclusive: at most one of the fields may be present + # + # "Present" means the key is given with a non-nil value (the same + # presence JSON Schema's dependencies keyword speaks about). def validate_rules!(inputs) rules = @specification.respond_to?(:rules) ? @specification.rules : nil return if rules.nil? || rules.empty? @@ -78,6 +88,9 @@ def validate_rules!(inputs) check, message, fields = if definition.respond_to?(:call) [definition, key.to_s, []] + elsif definition[:relation] + [RelationRules.check(definition), definition[:message] || RelationRules.message(definition), + definition[:fields] || []] else [definition.fetch(:check), definition[:message] || key.to_s, definition[:fields] || []] end diff --git a/lib/agentic/execution_journal.rb b/lib/agentic/execution_journal.rb index 9770484..3874427 100644 --- a/lib/agentic/execution_journal.rb +++ b/lib/agentic/execution_journal.rb @@ -23,7 +23,7 @@ module Agentic class ExecutionJournal # Replayed journal state: everything a resuming process needs to know ReplayedState = Struct.new( - :plan_id, :status, :completed_task_ids, :failed_task_ids, :outputs, :failures, :events, :descriptions, :durations, + :plan_id, :status, :completed_task_ids, :failed_task_ids, :outputs, :failures, :events, :descriptions, :durations, :duration_samples, keyword_init: true ) do # Task durations keyed by description - the natural baseline source @@ -33,6 +33,24 @@ def durations_by_description durations end + # A percentile over a task's recorded durations, for baselines that + # resist single-run noise (a journal may hold many runs) + # @param description [String] The task's description + # @param percentile [Numeric] 0-100 + # @param last [Integer, nil] Consider only the most recent N samples + # @return [Float, nil] The percentile duration, or nil if unrecorded + def duration_percentile(description, percentile, last: nil) + samples = duration_samples[description] + return nil if samples.nil? || samples.empty? + + samples = samples.last(last) if last + sorted = samples.sort + rank = (percentile / 100.0) * (sorted.size - 1) + lower = sorted[rank.floor] + upper = sorted[rank.ceil] + lower + (upper - lower) * (rank - rank.floor) + end + # @param key [String] A task id or a task description (descriptions # act as idempotency keys across runs, since ids are per-run) # @return [Boolean] True if the journal records a success for the task @@ -70,7 +88,9 @@ def lifecycle_hooks(hooks = {}) record(:task_succeeded, task_id: task_id, description: task.description, duration: duration, output: result.output) end, after_task_failure: chain(hooks[:after_task_failure]) do |task_id:, task:, failure:, duration:| - record(:task_failed, task_id: task_id, description: task.description, duration: duration, error: failure.message, error_type: failure.type) + record(:task_failed, task_id: task_id, description: task.description, duration: duration, + error: failure.message, error_type: failure.type, + retryable: failure.respond_to?(:retryable?) ? failure.retryable? : nil) end, plan_completed: chain(hooks[:plan_completed]) do |plan_id:, status:, execution_time:, tasks:, results:| record(:plan_completed, plan_id: plan_id, status: status, execution_time: execution_time) @@ -108,7 +128,8 @@ def self.replay(path:) failures: {}, events: [], descriptions: {}, - durations: {} + durations: {}, + duration_samples: Hash.new { |h, k| h[k] = [] } ) return state unless File.exist?(path) @@ -132,12 +153,13 @@ def self.replay(path:) state.outputs[task_id] = entry[:output] if entry[:description] && entry[:duration] state.durations[entry[:description]] = entry[:duration] + state.duration_samples[entry[:description]] << entry[:duration] end when "task_failed" task_id = entry[:task_id] unless state.completed_task_ids.include?(task_id) state.failed_task_ids << task_id unless state.failed_task_ids.include?(task_id) - state.failures[task_id] = {message: entry[:error], type: entry[:error_type]} + state.failures[task_id] = {message: entry[:error], type: entry[:error_type], retryable: entry[:retryable]} end when "plan_completed" state.plan_id = entry[:plan_id] diff --git a/lib/agentic/plan_orchestrator.rb b/lib/agentic/plan_orchestrator.rb index fc0604e..6d3277d 100644 --- a/lib/agentic/plan_orchestrator.rb +++ b/lib/agentic/plan_orchestrator.rb @@ -126,7 +126,9 @@ def graph stats: { depth: depth.freeze, max_depth: depth.values.max || 0, - max_fan_in: @dependencies.values.map(&:size).max || 0 + max_fan_in: @dependencies.values.map(&:size).max || 0, + roots: @dependencies.select { |_, deps| deps.empty? }.keys.freeze, + leaves: (@dependencies.keys - @dependencies.values.flatten).freeze }.freeze }.freeze end diff --git a/lib/agentic/rate_limit.rb b/lib/agentic/rate_limit.rb index 28aa9b2..6bc912e 100644 --- a/lib/agentic/rate_limit.rb +++ b/lib/agentic/rate_limit.rb @@ -54,6 +54,24 @@ def acquire # @return [Integer] Acquisitions currently inside the ceiling attr_reader :in_flight + # Changes the ceiling while the limiter is live - the seam adaptive + # throttles steer. Raising a concurrency ceiling resumes waiters + # immediately; lowering it lets in-flight work finish and admits + # nothing new until the count is back under the mark. Windowed + # limits apply the new ceiling on the next admission check. + # + # @param ceiling [Integer] The new maximum (must be positive) + # @return [Integer] The new ceiling + def resize(ceiling) + unless ceiling.is_a?(Integer) && ceiling.positive? + raise ArgumentError, "ceiling must be a positive Integer, got #{ceiling.inspect}" + end + + @ceiling = ceiling + @semaphore.limit = ceiling if @semaphore + ceiling + end + # Composes this limit with another: the block runs only inside BOTH. # Production APIs usually enforce two laws at once - a billed quota # (window) and a connection limit (ceiling): diff --git a/lib/agentic/relation_rules.rb b/lib/agentic/relation_rules.rb new file mode 100644 index 0000000..97da800 --- /dev/null +++ b/lib/agentic/relation_rules.rb @@ -0,0 +1,65 @@ +# frozen_string_literal: true + +module Agentic + # Relation-typed cross-field rules: predicates declared as data. + # + # A structured rule with a relation: key names its predicate instead + # of hiding it in a lambda, so validators can enforce it, generators + # can satisfy it, schema exports can project it, and diff tools can + # compare two versions of it: + # + # rules: { + # fits: {relation: :sum_lte, fields: [:cpu, :memory], limit: 100}, + # customs: {relation: :requires, fields: [:express, :customs_code]}, + # one_auth: {relation: :mutually_exclusive, fields: [:api_key, :oauth_token]} + # } + # + # "Present" means the key is given with a non-nil value - the same + # presence JSON Schema's dependencies keyword speaks about. + module RelationRules + SUPPORTED = %i[sum_lte requires mutually_exclusive].freeze + + module_function + + # Builds the predicate a relation declaration describes + # @param definition [Hash] The rule definition ({relation:, fields:, ...}) + # @return [Proc] inputs -> Boolean + # @raise [ArgumentError] For an unknown relation + def check(definition) + fields = definition.fetch(:fields) + case definition[:relation] + when :sum_lte + limit = definition.fetch(:limit) + ->(inputs) { fields.sum { |field| inputs[field] || 0 } <= limit } + when :requires + trigger, *needed = fields + ->(inputs) { !present?(inputs, trigger) || needed.all? { |field| present?(inputs, field) } } + when :mutually_exclusive + ->(inputs) { fields.count { |field| present?(inputs, field) } <= 1 } + else + raise ArgumentError, "unknown rule relation #{definition[:relation].inspect} " \ + "(supported: #{SUPPORTED.map(&:inspect).join(", ")})" + end + end + + # A human message derived from the declaration itself, used when the + # rule doesn't supply one + # @param definition [Hash] The rule definition + # @return [String] + def message(definition) + fields = definition.fetch(:fields) + case definition[:relation] + when :sum_lte then "#{fields.join(" + ")} must total at most #{definition[:limit]}" + when :requires then "#{fields.first} requires #{fields.drop(1).join(", ")}" + when :mutually_exclusive then "at most one of #{fields.join(", ")} may be given" + end + end + + # @param inputs [Hash] Symbolized inputs + # @param field [Symbol] The field to test + # @return [Boolean] True when the key is given with a non-nil value + def present?(inputs, field) + inputs.key?(field) && !inputs[field].nil? + end + end +end diff --git a/lib/agentic/task_failure.rb b/lib/agentic/task_failure.rb index 87639f0..2ca6df3 100644 --- a/lib/agentic/task_failure.rb +++ b/lib/agentic/task_failure.rb @@ -33,6 +33,21 @@ def retryable? @retryable end + # The nil convention: only an EXPLICIT false verdict is hopeless. + # An error that expressed no opinion (nil) gets suspicion, not a + # death sentence - breakers should count strikes against it, not + # trip instantly. These two predicates split the three-valued + # verdict at the joint policy code actually cares about. + # @return [Boolean] True when the error testified retrying can never help + def hopeless? + @retryable == false + end + + # @return [Boolean] True when retrying might help (verdict true or no opinion) + def possibly_transient? + !hopeless? + end + # Returns a serializable representation of the failure # @return [Hash] The failure as a hash def to_h diff --git a/spec/agentic/round10_features_spec.rb b/spec/agentic/round10_features_spec.rb new file mode 100644 index 0000000..a406dda --- /dev/null +++ b/spec/agentic/round10_features_spec.rb @@ -0,0 +1,98 @@ +# frozen_string_literal: true + +require "spec_helper" + +RSpec.describe "round 10 framework features" do + describe "relation-typed rules" do + let(:spec) do + Agentic::CapabilitySpecification.new( + name: "provision", description: "Provision a box", version: "1.0.0", + inputs: { + cpu: {type: "number", required: true}, + memory: {type: "number", required: true}, + express: {type: "boolean"}, + customs_code: {type: "string"}, + api_key: {type: "string"}, + oauth_token: {type: "string"} + }, + rules: { + fits: {relation: :sum_lte, fields: [:cpu, :memory], limit: 100}, + customs: {relation: :requires, fields: [:express, :customs_code]}, + one_auth: {relation: :mutually_exclusive, fields: [:api_key, :oauth_token]} + } + ) + end + let(:validator) { Agentic::CapabilityValidator.new(spec) } + + it "enforces sum_lte with a derived message" do + expect { + validator.validate_inputs!(cpu: 60, memory: 60) + }.to raise_error(Agentic::Errors::ValidationError) { |error| + violation = error.rule_violations.find { |v| v[:rule] == :fits } + expect(violation[:message]).to eq("cpu + memory must total at most 100") + expect(violation[:fields]).to eq([:cpu, :memory]) + } + + expect { validator.validate_inputs!(cpu: 40, memory: 60) }.not_to raise_error + end + + it "enforces requires only when the trigger is present" do + expect { + validator.validate_inputs!(cpu: 1, memory: 1, express: true) + }.to raise_error(Agentic::Errors::ValidationError, /express requires customs_code/) + + expect { + validator.validate_inputs!(cpu: 1, memory: 1, express: true, customs_code: "HS-1") + }.not_to raise_error + expect { validator.validate_inputs!(cpu: 1, memory: 1) }.not_to raise_error + end + + it "enforces mutual exclusion by presence" do + expect { + validator.validate_inputs!(cpu: 1, memory: 1, api_key: "k", oauth_token: "t") + }.to raise_error(Agentic::Errors::ValidationError, /at most one of api_key, oauth_token/) + + expect { validator.validate_inputs!(cpu: 1, memory: 1, api_key: "k") }.not_to raise_error + end + + it "rejects unknown relations loudly" do + bad = Agentic::CapabilitySpecification.new( + name: "x", description: "x", version: "1.0.0", + inputs: {a: {type: "number", required: true}}, + rules: {odd: {relation: :sum_gte, fields: [:a], limit: 1}} + ) + + expect { + Agentic::CapabilityValidator.new(bad).validate_inputs!(a: 1) + }.to raise_error(ArgumentError, /unknown rule relation :sum_gte/) + end + + it "projects expressible relations into draft-07 keywords and x-agentic-rules" do + schema = spec.to_json_schema + + expect(schema["dependencies"]).to eq({"express" => ["customs_code"]}) + expect(schema["allOf"]).to include({"not" => {"required" => %w[api_key oauth_token]}}) + + fits = schema["x-agentic-rules"].find { |r| r["rule"] == "fits" } + expect(fits["relation"]).to eq("sum_lte") + expect(fits["limit"]).to eq(100) + expect(fits["message"]).to eq("cpu + memory must total at most 100") + end + end + + describe "the retryable nil convention" do + it "treats only an explicit false as hopeless" do + hopeless = Agentic::TaskFailure.from_exception(Agentic::Errors::LlmAuthenticationError.new("401")) + transient = Agentic::TaskFailure.from_exception(Agentic::Errors::LlmRateLimitError.new("429")) + no_opinion = Agentic::TaskFailure.from_exception(RuntimeError.new("boom")) + + expect(hopeless.hopeless?).to be(true) + expect(hopeless.possibly_transient?).to be(false) + expect(transient.hopeless?).to be(false) + expect(transient.possibly_transient?).to be(true) + expect(no_opinion.retryable?).to be_nil + expect(no_opinion.hopeless?).to be(false) + expect(no_opinion.possibly_transient?).to be(true) + end + end +end diff --git a/spec/agentic/round8_features_spec.rb b/spec/agentic/round8_features_spec.rb new file mode 100644 index 0000000..d1e8f44 --- /dev/null +++ b/spec/agentic/round8_features_spec.rb @@ -0,0 +1,74 @@ +# frozen_string_literal: true + +require "spec_helper" +require "tmpdir" + +RSpec.describe "round 8 framework features" do + def task_named(description) + Agentic::Task.new( + description: description, + agent_spec: {"name" => "worker", "instructions" => "work"} + ) + end + + describe "stats[:roots] and stats[:leaves]" do + it "identifies tasks with no dependencies and tasks nothing depends on" do + orchestrator = Agentic::PlanOrchestrator.new + root_a = task_named("root-a") + root_b = task_named("root-b") + middle = task_named("middle") + leaf = task_named("leaf") + + orchestrator.add_task(root_a, agent: ->(_t) { :ok }) + orchestrator.add_task(root_b, agent: ->(_t) { :ok }) + orchestrator.add_task(middle, [root_a, root_b], agent: ->(_t) { :ok }) + orchestrator.add_task(leaf, [middle], agent: ->(_t) { :ok }) + + stats = orchestrator.graph[:stats] + + expect(stats[:roots]).to contain_exactly(root_a.id, root_b.id) + expect(stats[:leaves]).to eq([leaf.id]) + end + end + + describe "journal duration percentiles" do + it "computes percentiles over accumulated samples across runs" do + Dir.mktmpdir do |dir| + path = File.join(dir, "history.jsonl") + journal = Agentic::ExecutionJournal.new(path: path) + + [0.01, 0.02, 0.03, 0.04, 0.10].each_with_index do |duration, i| + journal.record(:task_succeeded, task_id: "run#{i}", description: "nightly", duration: duration, output: nil) + end + + state = Agentic::ExecutionJournal.replay(path: path) + + expect(state.duration_samples["nightly"].size).to eq(5) + expect(state.duration_percentile("nightly", 50)).to be_within(0.001).of(0.03) + expect(state.duration_percentile("nightly", 100)).to eq(0.10) + expect(state.duration_percentile("nightly", 50, last: 2)).to be_within(0.001).of(0.07) + expect(state.duration_percentile("unknown", 50)).to be_nil + end + end + end + + describe "x-agentic-rules in the schema export" do + it "emits structured rule metadata and omits prose lambdas" do + spec = Agentic::CapabilitySpecification.new( + name: "quote", description: "quotes", version: "1.0.0", + inputs: {mode: {type: "string", required: true}}, + rules: { + :air_limit => {message: "air max 500kg", fields: [:mode], check: ->(_i) { true }}, + "prose rule" => ->(_i) { true } + } + ) + + schema = spec.to_json_schema + + expect(schema["x-agentic-rules"]).to eq([ + {"rule" => "air_limit", "message" => "air max 500kg", "fields" => ["mode"]} + ]) + expect(spec.to_json_schema(:outputs)).not_to have_key("x-agentic-rules") + end + end +end diff --git a/spec/agentic/round9_features_spec.rb b/spec/agentic/round9_features_spec.rb new file mode 100644 index 0000000..f2c7664 --- /dev/null +++ b/spec/agentic/round9_features_spec.rb @@ -0,0 +1,85 @@ +# frozen_string_literal: true + +require "spec_helper" +require "tmpdir" + +RSpec.describe "round 9 framework features" do + describe "RateLimit#resize" do + it "raises a concurrency ceiling while the limiter is live" do + limit = Agentic::RateLimit.new(1) + + Sync do + 4.times.map { Async { limit.acquire { sleep(0.01) } } }.each(&:wait) + expect(limit.high_water).to eq(1) + + limit.resize(3) + 6.times.map { Async { limit.acquire { sleep(0.01) } } }.each(&:wait) + end + + expect(limit.ceiling).to eq(3) + expect(limit.high_water).to eq(3) + end + + it "lowers a windowed ceiling for subsequent admissions" do + limit = Agentic::RateLimit.new(5, per: 0.1) + limit.resize(2) + stamps = [] + + Sync do + 3.times.map { + Async do + limit.acquire { stamps << Process.clock_gettime(Process::CLOCK_MONOTONIC) } + end + }.each(&:wait) + end + + # With the resized ceiling of 2, the 3rd admission waits a window + windows = stamps.sort + expect(windows[2] - windows[0]).to be >= 0.09 + end + + it "rejects non-positive and non-integer ceilings" do + limit = Agentic::RateLimit.new(2) + + expect { limit.resize(0) }.to raise_error(ArgumentError, /positive Integer/) + expect { limit.resize(2.5) }.to raise_error(ArgumentError, /positive Integer/) + expect(limit.ceiling).to eq(2) + end + end + + describe "journaled retryable verdicts" do + it "records the failure's own retryability at write time and replays it" do + Dir.mktmpdir do |dir| + path = File.join(dir, "run.journal.jsonl") + journal = Agentic::ExecutionJournal.new(path: path) + hooks = journal.lifecycle_hooks + task = Agentic::Task.new(description: "sync", agent_spec: {"name" => "w", "instructions" => "work"}) + + transient = Agentic::TaskFailure.from_exception(Agentic::Errors::LlmRateLimitError.new("429")) + hooks[:after_task_failure].call(task_id: "t1", task: task, failure: transient, duration: 0.1) + + state = Agentic::ExecutionJournal.replay(path: path) + + expect(state.events.last[:retryable]).to be(true) + expect(state.failures["t1"][:retryable]).to be(true) + end + end + + it "preserves a nil verdict when the error expressed no opinion" do + Dir.mktmpdir do |dir| + path = File.join(dir, "run.journal.jsonl") + journal = Agentic::ExecutionJournal.new(path: path) + hooks = journal.lifecycle_hooks + task = Agentic::Task.new(description: "sync", agent_spec: {"name" => "w", "instructions" => "work"}) + + opinionless = Agentic::TaskFailure.from_exception(RuntimeError.new("boom")) + hooks[:after_task_failure].call(task_id: "t1", task: task, failure: opinionless, duration: 0.1) + + state = Agentic::ExecutionJournal.replay(path: path) + + expect(state.events.last).to have_key(:retryable) + expect(state.failures["t1"][:retryable]).to be_nil + end + end + end +end