From e5ecca764ec814c0b56295df56248e7a5d8872b2 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 6 Jul 2026 21:21:23 +0000 Subject: [PATCH 01/36] feat: deliver the round-4 asks The prioritized asks from the round-4 field notes, implemented: - PlanOrchestrator#graph: frozen read-only snapshot of the plan's topology (tasks, dependencies, named needs) for tools that render, review, or analyze without crowbarring ivars - ValidationError#expectations carries the declared contract for each violated key, so errors can name what would have been legal - Cross-field contract rules: CapabilitySpecification rules: maps a description to a predicate over the whole input; broken rules are reported together under :base after per-key validation - Agentic::RateLimit: credential-scoped concurrency ceiling with a high-water mark, shareable across plans; LlmClient accepts limiter: - Retry backoff jitter defaults ON (floored at zero) so fleets don't retry in lockstep The three examples that asked for these are modernized onto them: graph_critic reads #graph, state_machine renders legal states from expectations, shared_rate_limit uses Agentic::RateLimit. README verifier confirms the new documentation snippets parse. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01FeZLdZ3M4q4cho4DZPfSkF --- README.md | 20 ++- examples/graph_critic.rb | 9 +- examples/shared_rate_limit.rb | 17 +-- examples/state_machine.rb | 4 +- lib/agentic/capability_specification.rb | 11 +- lib/agentic/capability_validator.rb | 33 ++++- lib/agentic/errors.rb | 10 +- lib/agentic/llm_client.rb | 12 +- lib/agentic/plan_orchestrator.rb | 24 +++- lib/agentic/rate_limit.rb | 51 +++++++ spec/agentic/plan_orchestrator_config_spec.rb | 3 +- spec/agentic/round5_features_spec.rb | 124 ++++++++++++++++++ 12 files changed, 284 insertions(+), 34 deletions(-) create mode 100644 lib/agentic/rate_limit.rb create mode 100644 spec/agentic/round5_features_spec.rb diff --git a/README.md b/README.md index b7aaa2d..f0fdf98 100644 --- a/README.md +++ b/README.md @@ -234,11 +234,21 @@ orchestrator.add_task(next_verse, [previous_verse], agent: ->(task) { ``` Capability contracts can constrain values beyond type and presence — -`enum: %w[standard express]`, `min:`/`max:` bounds, and `non_empty: true` -for strings and arrays. The `task_slot_acquired` lifecycle hook fires when -a task obtains a concurrency slot (with `waited:` time), separating queue -wait from run time; and retry policies consult an error's own -`retryable?` verdict before falling back to the `retryable_errors` list. +`enum: %w[standard express]`, `min:`/`max:` bounds, `non_empty: true` for +strings and arrays, and cross-field `rules:` checked over the whole input +(`rules: {"express max 10 items" => ->(i) { i[:speed] != "express" || i[:quantity] <= 10 }}`). +`ValidationError#expectations` carries the declared contract for each +violated key, so error messages can name what would have been legal. + +The `task_slot_acquired` lifecycle hook fires when a task obtains a +concurrency slot (with `waited:` time), separating queue wait from run +time; retry policies consult an error's own `retryable?` verdict before +falling back to the `retryable_errors` list, and backoff jitter is on by +default so fleets don't retry in lockstep. `PlanOrchestrator#graph` +returns a frozen snapshot of the plan's topology for tools that render or +review it, and `Agentic::RateLimit` is a credential-scoped concurrency +ceiling shareable across plans and clients +(`Agentic::LlmClient.new(config, limiter: rate_limit)`). ### The concurrency contract diff --git a/examples/graph_critic.rb b/examples/graph_critic.rb index c48b1e2..31161c8 100644 --- a/examples/graph_critic.rb +++ b/examples/graph_critic.rb @@ -40,10 +40,11 @@ def task_named(name) orchestrator.add_task(tasks["lonely"]) # --- the critique ---------------------------------------------------------- -# The graph is private, so the critic borrows a crowbar. A read-only -# graph view on the orchestrator is this example's feature request. -dependencies = orchestrator.instance_variable_get(:@dependencies) -names = orchestrator.tasks.transform_values(&:description) +# This example's original feature request, granted: a read-only view of +# the plan's topology. No more crowbar. +graph = orchestrator.graph +dependencies = graph[:dependencies] +names = graph[:tasks].transform_values(&:description) dependents = Hash.new { |h, k| h[k] = [] } dependencies.each { |task_id, deps| deps.each { |dep| dependents[dep] << task_id } } diff --git a/examples/shared_rate_limit.rb b/examples/shared_rate_limit.rb index 085137e..6235e81 100644 --- a/examples/shared_rate_limit.rb +++ b/examples/shared_rate_limit.rb @@ -12,28 +12,25 @@ require_relative "../lib/agentic" require "async" -require "async/semaphore" API_CEILING = 3 -# The shared credential: a semaphore plus a high-water-mark counter +# The shared credential: Agentic::RateLimit (this example's original +# feature request, granted) plus a call log for the interleaving proof class RateLimitedApi - attr_reader :high_water + attr_reader :limit def initialize(ceiling) - @semaphore = Async::Semaphore.new(ceiling) - @in_flight = 0 - @high_water = 0 + @limit = Agentic::RateLimit.new(ceiling) @calls = [] end + def high_water = @limit.high_water + def call(plan, name, latency) - @semaphore.acquire do - @in_flight += 1 - @high_water = [@high_water, @in_flight].max + @limit.acquire do @calls << "#{plan}/#{name}" sleep(latency) - @in_flight -= 1 "#{name}:ok" end end diff --git a/examples/state_machine.rb b/examples/state_machine.rb index 2d114d2..e617dbf 100644 --- a/examples/state_machine.rb +++ b/examples/state_machine.rb @@ -57,7 +57,9 @@ def fire(event) @history << @state [:ok, @state] rescue Agentic::Errors::ValidationError => e - allowed = TRANSITIONS.fetch(event)[:from] + # The violation now carries the contract's expectation - no + # side-channel lookup into the transition table needed + allowed = e.expectations.dig(:state, :enum) || [] [:illegal, "cannot #{event} from '#{@state}' (legal from: #{allowed.join(", ")}) - #{e.violations.keys.join(", ")} violated"] end end diff --git a/lib/agentic/capability_specification.rb b/lib/agentic/capability_specification.rb index 756173f..bf6e356 100644 --- a/lib/agentic/capability_specification.rb +++ b/lib/agentic/capability_specification.rb @@ -18,15 +18,24 @@ class CapabilitySpecification # @param inputs [Hash] The required inputs for the capability # @param outputs [Hash] The expected outputs from the capability # @param dependencies [Array] The dependencies of the capability - def initialize(name:, description:, version:, inputs: {}, outputs: {}, dependencies: []) + def initialize(name:, description:, version:, inputs: {}, outputs: {}, dependencies: [], rules: {}) @name = name @description = description @version = version @inputs = inputs @outputs = outputs @dependencies = dependencies + @rules = rules end + # Cross-field input rules: description => predicate over the full + # (symbolized) inputs hash. Checked after per-key validation passes: + # + # rules: {"express orders max 10 items" => ->(i) { i[:speed] != "express" || i[:quantity] <= 10 }} + # + # @return [Hash{String=>#call}] + attr_reader :rules + # Check if this capability is compatible with another capability # @param other [CapabilitySpecification] The other capability # @return [Boolean] True if compatible diff --git a/lib/agentic/capability_validator.rb b/lib/agentic/capability_validator.rb index 972aa25..2429554 100644 --- a/lib/agentic/capability_validator.rb +++ b/lib/agentic/capability_validator.rb @@ -42,15 +42,38 @@ def validate_outputs!(outputs) private def validate!(kind, declared, values) - return if declared.nil? || declared.empty? + symbolized = symbolize_keys(values) - result = schema_for(kind, declared).call(symbolize_keys(values)) - return if result.success? + if declared && !declared.empty? + result = schema_for(kind, declared).call(symbolized) + + unless result.success? + violations = result.errors.to_h + raise Errors::ValidationError.new( + capability: @specification.name, + kind: kind, + violations: violations, + expectations: declared.slice(*violations.keys) + ) + end + end + + validate_rules!(symbolized) if kind == :inputs + end + + # Cross-field rules run after per-key validation, over the whole + # inputs hash; every broken rule is reported at once under :base + def validate_rules!(inputs) + rules = @specification.respond_to?(:rules) ? @specification.rules : nil + return if rules.nil? || rules.empty? + + broken = rules.reject { |_description, check| check.call(inputs) }.keys + return if broken.empty? raise Errors::ValidationError.new( capability: @specification.name, - kind: kind, - violations: result.errors.to_h + kind: :inputs, + violations: {base: broken} ) end diff --git a/lib/agentic/errors.rb b/lib/agentic/errors.rb index c4b1ec9..643f0da 100644 --- a/lib/agentic/errors.rb +++ b/lib/agentic/errors.rb @@ -26,13 +26,21 @@ class ValidationError < StandardError # @return [Hash{Symbol=>Array}] Violation messages keyed by attribute attr_reader :violations + # The declared contract for each violated key - type, enum, min/max, + # required - so callers can render what WOULD have been legal without + # side-channel knowledge of the contract + # @return [Hash{Symbol=>Hash}] Declarations keyed by violated attribute + attr_reader :expectations + # @param capability [String] The capability name # @param kind [Symbol] :inputs or :outputs # @param violations [Hash{Symbol=>Array}] Messages keyed by attribute - def initialize(capability:, kind:, violations:) + # @param expectations [Hash{Symbol=>Hash}] Declarations for violated keys + def initialize(capability:, kind:, violations:, expectations: {}) @capability = capability @kind = kind @violations = violations + @expectations = expectations details = violations.map { |key, messages| "#{key} #{Array(messages).join(", ")}" }.join("; ") super("Invalid #{kind} for capability '#{capability}': #{details}") diff --git a/lib/agentic/llm_client.rb b/lib/agentic/llm_client.rb index e069791..66c91b9 100644 --- a/lib/agentic/llm_client.rb +++ b/lib/agentic/llm_client.rb @@ -15,9 +15,12 @@ class LlmClient # Initializes a new LlmClient # @param config [LlmConfig] The configuration for the LLM # @param retry_config [RetryConfig, Hash] Configuration for the retry handler - def initialize(config, retry_config = {}) + # @param limiter [RateLimit, nil] A credential-scoped concurrency ceiling; + # share one limiter across every client using the same API key + def initialize(config, retry_config = {}, limiter: nil) configuration = Agentic.configuration configuration.validate! + @limiter = limiter # Local endpoints (Ollama, etc.) ignore the token but the client # requires one, so send an explicit placeholder rather than nil @@ -63,7 +66,12 @@ def complete(messages, output_schema: nil, fail_on_error: false, use_retries: tr parameters.merge!(options) execution_method = use_retries ? method(:with_retry) : method(:without_retry) - execution_method.call(messages, parameters, output_schema, fail_on_error) + + if @limiter + @limiter.acquire { execution_method.call(messages, parameters, output_schema, fail_on_error) } + else + execution_method.call(messages, parameters, output_schema, fail_on_error) + end end # Executes the API call with retries for transient errors diff --git a/lib/agentic/plan_orchestrator.rb b/lib/agentic/plan_orchestrator.rb index 090b4d7..3c33da0 100644 --- a/lib/agentic/plan_orchestrator.rb +++ b/lib/agentic/plan_orchestrator.rb @@ -38,11 +38,15 @@ def initialize(plan_id: SecureRandom.uuid, concurrency_limit: 10, retry_policy: @task_agents = {} @task_needs = {} - # Configure retry policy with defaults + # Configure retry policy with defaults. Jitter defaults ON: a fleet + # retrying an upstream on the same schedule is a synchronized + # stampede; pass backoff_jitter: false to opt out (e.g. in tests + # that assert exact delays) @retry_policy = { max_retries: 3, retryable_errors: ["TimeoutError"], - backoff_strategy: :constant + backoff_strategy: :constant, + backoff_jitter: true }.merge(retry_policy) # Configure lifecycle hooks with callable defaults (no-ops). @@ -90,6 +94,18 @@ def add_task(task, dependencies = [], agent: nil, needs: nil) @execution_state[:pending].add(task_id) end + # A read-only snapshot of the plan's topology, for tools that render, + # review, or analyze the graph without executing it + # @return [Hash] :tasks (id => Task), :dependencies (id => [ids]), + # :needs (id => {name => id}) + def graph + { + tasks: @tasks.dup.freeze, + dependencies: @dependencies.transform_values { |deps| deps.dup.freeze }.freeze, + needs: @task_needs.transform_values { |named| named.dup.freeze }.freeze + }.freeze + end + # Executes the plan, respecting task dependencies and concurrency limits # # Composes with structured concurrency: when called inside a running @@ -241,11 +257,11 @@ def apply_retry_backoff(task:) 0 end - # Apply jitter if configured + # Apply jitter (on by default) so fleets don't retry in lockstep if @retry_policy[:backoff_jitter] jitter_factor = 0.25 # Default 25% jitter jitter = rand(-delay * jitter_factor..delay * jitter_factor) - delay += jitter + delay = [delay + jitter, 0].max end # Sleep in the current task so the retry actually waits; the async diff --git a/lib/agentic/rate_limit.rb b/lib/agentic/rate_limit.rb new file mode 100644 index 0000000..cfa6587 --- /dev/null +++ b/lib/agentic/rate_limit.rb @@ -0,0 +1,51 @@ +# frozen_string_literal: true + +require "async/semaphore" + +module Agentic + # A credential-scoped concurrency ceiling, shareable across plans, + # clients, and agents in one process. Rate limits belong to the + # resource they protect - one API key means one ceiling, no matter + # how many orchestrators are running: + # + # limit = Agentic::RateLimit.new(3) + # client_a = Agentic::LlmClient.new(config, limiter: limit) + # client_b = Agentic::LlmClient.new(config, limiter: limit) + # # combined in-flight requests never exceed 3 + # + # The high-water mark records the most concurrent acquisitions ever + # observed, so tests and dashboards can verify the ceiling held. + class RateLimit + # @return [Integer] The maximum concurrent acquisitions allowed + attr_reader :ceiling + + # @return [Integer] The most concurrent acquisitions observed + attr_reader :high_water + + # @param ceiling [Integer] Maximum concurrent acquisitions + def initialize(ceiling) + @ceiling = ceiling + @semaphore = Async::Semaphore.new(ceiling) + @in_flight = 0 + @high_water = 0 + end + + # Runs the block inside the ceiling, waiting for a slot if necessary + # @yield The rate-limited work + # @return [Object] The block's return value + def acquire + @semaphore.acquire do + @in_flight += 1 + @high_water = [@high_water, @in_flight].max + begin + yield + ensure + @in_flight -= 1 + end + end + end + + # @return [Integer] Acquisitions currently inside the ceiling + attr_reader :in_flight + end +end diff --git a/spec/agentic/plan_orchestrator_config_spec.rb b/spec/agentic/plan_orchestrator_config_spec.rb index 6a3dae2..cacc4a8 100644 --- a/spec/agentic/plan_orchestrator_config_spec.rb +++ b/spec/agentic/plan_orchestrator_config_spec.rb @@ -46,7 +46,8 @@ expect(orchestrator.retry_policy[:backoff_base]).to eq(2) # Verify defaults are still present for options not specified - expect(orchestrator.retry_policy[:backoff_jitter]).to be_nil + # (jitter defaults ON so fleets don't retry in lockstep) + expect(orchestrator.retry_policy[:backoff_jitter]).to be true end it "merges custom policy with defaults" do diff --git a/spec/agentic/round5_features_spec.rb b/spec/agentic/round5_features_spec.rb new file mode 100644 index 0000000..ea9bda6 --- /dev/null +++ b/spec/agentic/round5_features_spec.rb @@ -0,0 +1,124 @@ +# frozen_string_literal: true + +require "spec_helper" + +RSpec.describe "round 5 framework features" do + def task_named(description) + Agentic::Task.new( + description: description, + agent_spec: {"name" => "worker", "instructions" => "work"} + ) + end + + describe "PlanOrchestrator#graph" do + it "exposes a frozen read-only snapshot of the topology" do + orchestrator = Agentic::PlanOrchestrator.new + fetch = task_named("fetch") + merge = task_named("merge") + orchestrator.add_task(fetch, agent: ->(_t) { :ok }) + orchestrator.add_task(merge, [fetch], needs: {source: fetch}, agent: ->(_t) { :ok }) + + graph = orchestrator.graph + + expect(graph[:tasks].keys).to contain_exactly(fetch.id, merge.id) + expect(graph[:dependencies][merge.id]).to eq([fetch.id]) + expect(graph[:needs][merge.id]).to eq(source: fetch.id) + expect(graph).to be_frozen + expect(graph[:dependencies]).to be_frozen + expect { graph[:dependencies][merge.id] << "x" }.to raise_error(FrozenError) + end + end + + describe "ValidationError#expectations" do + let(:specification) do + Agentic::CapabilitySpecification.new( + name: "ship", description: "ships", version: "1.0.0", + inputs: { + speed: {type: "string", required: true, enum: %w[standard express]}, + quantity: {type: "number", required: true, min: 1} + } + ) + end + + it "carries the declared contract for each violated key" do + validator = Agentic::CapabilityValidator.new(specification) + + expect { + validator.validate_inputs!(speed: "teleport", quantity: 5) + }.to raise_error(Agentic::Errors::ValidationError) { |error| + expect(error.expectations.keys).to eq([:speed]) + expect(error.expectations[:speed][:enum]).to eq(%w[standard express]) + } + end + end + + describe "cross-field contract rules" do + let(:specification) do + Agentic::CapabilitySpecification.new( + name: "quote", description: "quotes freight", version: "1.0.0", + inputs: { + speed: {type: "string", required: true, enum: %w[standard express]}, + quantity: {type: "number", required: true, min: 1} + }, + rules: { + "express orders are limited to 10 items" => ->(i) { i[:speed] != "express" || i[:quantity] <= 10 } + } + ) + end + + it "passes inputs that satisfy every rule" do + validator = Agentic::CapabilityValidator.new(specification) + + expect { validator.validate_inputs!(speed: "express", quantity: 10) }.not_to raise_error + expect { validator.validate_inputs!(speed: "standard", quantity: 500) }.not_to raise_error + end + + it "reports broken rules under :base after per-key validation passes" do + validator = Agentic::CapabilityValidator.new(specification) + + expect { + validator.validate_inputs!(speed: "express", quantity: 11) + }.to raise_error(Agentic::Errors::ValidationError) { |error| + expect(error.violations[:base]).to eq(["express orders are limited to 10 items"]) + } + end + end + + describe "RateLimit" do + it "bounds concurrent acquisitions across callers and records the high-water mark" do + limit = Agentic::RateLimit.new(2) + + Sync do |task| + 8.times.map { + task.async do + limit.acquire { sleep(0.01) } + end + }.each(&:wait) + end + + expect(limit.high_water).to eq(2) + expect(limit.in_flight).to eq(0) + end + + it "is accepted by LlmClient as limiter:" do + client = Agentic::LlmClient.new(Agentic::LlmConfig.new, limiter: Agentic::RateLimit.new(1)) + expect(client).to be_a(Agentic::LlmClient) + end + end + + describe "jitter default" do + it "defaults backoff_jitter on and never sleeps a negative delay" do + orchestrator = Agentic::PlanOrchestrator.new(retry_policy: {backoff_strategy: :constant, backoff_constant: 0.01}) + slept = [] + allow(orchestrator).to receive(:sleep) { |delay| slept << delay } + + task = task_named("flaky") + task.retry_count = 1 + 5.times { orchestrator.apply_retry_backoff(task: task) } + + expect(orchestrator.retry_policy[:backoff_jitter]).to be true + expect(slept).to all(be >= 0) + expect(slept.uniq.size).to be > 1 # jitter actually varies the delay + end + end +end From c99bb3aacede2cce375f4b49dbab8aafc8d1298d Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 6 Jul 2026 21:22:32 +0000 Subject: [PATCH 02/36] docs: dungeon crawl example (Matz round 5) A quest as a plan: the map is drawn from orchestrator.graph before the run, so the document and the program are one truth. Field notes ask for a topological order in the graph snapshot. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01FeZLdZ3M4q4cho4DZPfSkF --- docs/perspectives/round-5/01-matz.md | 54 ++++++++++++++++++++ examples/dungeon_crawl.rb | 74 ++++++++++++++++++++++++++++ 2 files changed, 128 insertions(+) create mode 100644 docs/perspectives/round-5/01-matz.md create mode 100644 examples/dungeon_crawl.rb diff --git a/docs/perspectives/round-5/01-matz.md b/docs/perspectives/round-5/01-matz.md new file mode 100644 index 0000000..baf9d82 --- /dev/null +++ b/docs/perspectives/round-5/01-matz.md @@ -0,0 +1,54 @@ +# Round 5 field notes — Matz maps the dungeon + +*Built: `examples/dungeon_crawl.rb` — a quest as a plan: rooms are +tasks, doors are dependencies, and the map is drawn from +`orchestrator.graph` before anyone delves.* + +## What I built and why + +Every dungeon game maintains two truths: the map the player sees and +the graph the engine walks — and every dungeon game eventually ships a +bug where they disagree. This crawl has one truth. The map is printed +from `orchestrator.graph`, the same frozen topology the scheduler will +execute: + +``` +[Entrance Hall] <- you are here +[Treasury] doors from: Spider Nest, Flooded Crypt +``` + +Then the party delves — nest and crypt in parallel, the treasury +waiting on both keys via `needs: {web:, depths:}` — and the loot fans +in: "a chest of coppers (unlocked with someone's boot and a silver +coin)." There is no second map to fall out of date, because the +document and the program are the same object. That is the deepest kind +of DRY: not avoiding repeated *text*, but avoiding repeated *truth*. + +## What pleased me + +- `graph` being **frozen** is the right manner. A map you can scribble + on is a map you can lie with; the accessor hands you a photograph, + not the territory's steering wheel. (I tried to mutate it, for + science. `FrozenError`. Good.) +- Five features from three rounds cohabit in sixty lines without + crowding: payloads, callables, positional deps, named needs, the + graph view. New vocabulary that doesn't jostle the old vocabulary is + the sign the design is growing rather than accreting. +- Seeded loot, again. "Reproducible whimsy" is my favorite genre now. + +## A small observation for the maintainers + +The map loop wants the rooms in *dependency order* (entrance first, +treasury last), and I got it by accident because insertion order +matched. `graph` preserves insertion order — Ruby hashes promise that — +but a plan built in scrambled order would print a scrambled map. A +`graph[:order]` with a topological sort would let map-drawers be +correct on purpose instead of lucky. (Aaron will want it for critical +paths within the hour; ask him.) + +## Verdict + +The framework can now draw itself before it runs itself. When a +library's introspection is good enough to build a game's UI from, the +architecture documents can retire — the code has learned to give the +tour. diff --git a/examples/dungeon_crawl.rb b/examples/dungeon_crawl.rb new file mode 100644 index 0000000..1caec78 --- /dev/null +++ b/examples/dungeon_crawl.rb @@ -0,0 +1,74 @@ +# frozen_string_literal: true + +# The Dungeon Crawl: a quest is a plan, rooms are tasks, and doors are +# dependencies. The map is drawn from the orchestrator's own graph +# BEFORE the run - then the party delves, and the treasure fans in. +# +# bundle exec ruby examples/dungeon_crawl.rb [seed] +# +# Runs offline. The dungeon is the dependency graph; there is no +# second map to fall out of date. + +require_relative "../lib/agentic" + +seed = (ARGV.first || 4).to_i +rng = Random.new(seed) + +LOOT = { + "Entrance Hall" => ["a rusty key", "a torch stub", "an ominous note"], + "Spider Nest" => ["silk rope", "a shed fang", "someone's boot"], + "Flooded Crypt" => ["a waterlogged tome", "a silver coin", "an eyeless fish"], + "Treasury" => ["the Amulet of Yendor", "a chest of coppers", "a suspicious goose"] +}.freeze + +orchestrator = Agentic::PlanOrchestrator.new(concurrency_limit: 2) + +rooms = {} +delve = ->(t) { + sleep(0.02) # every room takes delving + LOOT.fetch(t.description).sample(random: rng) +} + +def room(name) + Agentic::Task.new(description: name, agent_spec: {"name" => name, "instructions" => "delve"}) +end + +rooms["Entrance Hall"] = room("Entrance Hall") +rooms["Spider Nest"] = room("Spider Nest") +rooms["Flooded Crypt"] = room("Flooded Crypt") +rooms["Treasury"] = room("Treasury") + +orchestrator.add_task(rooms["Entrance Hall"], agent: delve) +orchestrator.add_task(rooms["Spider Nest"], [rooms["Entrance Hall"]], agent: delve) +orchestrator.add_task(rooms["Flooded Crypt"], [rooms["Entrance Hall"]], agent: delve) +orchestrator.add_task(rooms["Treasury"], + needs: {web: rooms["Spider Nest"], depths: rooms["Flooded Crypt"]}, + agent: ->(t) { + "#{LOOT.fetch(t.description).sample(random: rng)} (unlocked with #{t.needs.web} and #{t.needs.depths})" + }) + +# --- the map, drawn from the plan itself ------------------------------------ +graph = orchestrator.graph +names = graph[:tasks].transform_values(&:description) + +puts "THE MAP (drawn from orchestrator.graph, before anyone delves)" +puts +graph[:dependencies].each do |room_id, door_ids| + if door_ids.empty? + puts " [#{names[room_id]}] <- you are here" + else + puts " [#{names[room_id]}] doors from: #{door_ids.map { |d| names[d] }.join(", ")}" + end +end + +# --- the delve --------------------------------------------------------------- +result = orchestrator.execute_plan + +puts +puts "THE DELVE (seed #{seed})" +rooms.each do |name, task| + puts " #{name}: found #{result.results[task.id].output}" +end +puts +puts "(#{result.status} in #{(result.execution_time * 1000).round}ms - the nest and" +puts " the crypt were delved in parallel; the treasury needed both keys)" From 0c25c8718866b801c2cf5dab0c5c48ef7e567c0c Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 6 Jul 2026 21:23:35 +0000 Subject: [PATCH 03/36] docs: live kanban board example (DHH round 5) A running plan rendered as To Do / Doing / Done from two lifecycle hooks; the WIP limit is the concurrency limit, enforced structurally. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01FeZLdZ3M4q4cho4DZPfSkF --- docs/perspectives/round-5/02-dhh.md | 49 +++++++++++++++++++ examples/kanban_board.rb | 73 +++++++++++++++++++++++++++++ 2 files changed, 122 insertions(+) create mode 100644 docs/perspectives/round-5/02-dhh.md create mode 100644 examples/kanban_board.rb diff --git a/docs/perspectives/round-5/02-dhh.md b/docs/perspectives/round-5/02-dhh.md new file mode 100644 index 0000000..6b94dba --- /dev/null +++ b/docs/perspectives/round-5/02-dhh.md @@ -0,0 +1,49 @@ +# Round 5 field notes — DHH ships the kanban board + +*Built: `examples/kanban_board.rb` — a running plan rendered as To Do / +Doing / Done, every frame captured live from lifecycle hooks.* + +## What I built and why + +We spent two decades wiring project-management tools to *approximate* +the state of work, staffed with people whose job is updating the +approximation. But when the work is a plan, the orchestrator already +IS the board — the columns are just `pending`, `in_progress`, and +`completed` wearing better clothes. Two hooks move the cards: + +- `task_slot_acquired` → card moves To Do → Doing (note: *slot* + acquired, not scheduled — a card in Doing means someone is actually + working it, which is the entire honesty proposition of a kanban + board; before round 4's hook this column would have lied) +- `after_task_success` → Doing → Done + +Twelve frames, 302ms, and mid-flight the board shows exactly what a +two-person team looks like: `layout page` in Doing, `review` and +`publish` waiting on it, three cards shipped. Nobody typed a status. + +## The insight worth the price of admission + +The **WIP limit is the concurrency limit.** Kanban's whole discipline — +"limit work in progress" — is `concurrency_limit: 2`, enforced by the +scheduler instead of by a coach reminding people in a meeting. When +your process tool and your execution engine are the same object, the +process can't drift from reality, because it *is* reality. That's the +argument I've been making about software writ small: the best process +is the one your system enforces structurally, invisibly, for free. + +## Notes + +- The board never shows a "Blocked" column because dependency-waiting + cards just stay in To Do. For this demo that's fine; a real board + wants `blocked` derived from graph[:dependencies] minus done. One + `orchestrator.graph` call — everything's there. Fizzy could be forty + lines on this framework, and I'm only half joking. +- Frame capture is an array push in a hook — Samuel's + "hooks-run-inline" contract means I thought about cost for exactly + one second, which is what a documented contract buys. + +## Verdict + +Five rounds ago this framework couldn't tell you what it was doing; +now it renders its own kanban in real time from two hooks. Delete the +status meeting. The plan will speak for itself. diff --git a/examples/kanban_board.rb b/examples/kanban_board.rb new file mode 100644 index 0000000..ca57aa1 --- /dev/null +++ b/examples/kanban_board.rb @@ -0,0 +1,73 @@ +# frozen_string_literal: true + +# The Kanban Board: a plan rendered as the three columns everyone +# actually understands - To Do, Doing, Done - reprinted at every state +# change while the plan runs. No standup, no sync meeting, no PM tool +# subscription: the orchestrator IS the board. +# +# bundle exec ruby examples/kanban_board.rb +# +# Runs offline; watch the cards move. + +require_relative "../lib/agentic" + +CARDS = { + "write copy" => 0.06, + "shoot photos" => 0.10, + "edit photos" => 0.05, + "layout page" => 0.08, + "review" => 0.04, + "publish" => 0.03 +}.freeze + +DEPS = { + "edit photos" => ["shoot photos"], + "layout page" => ["write copy", "edit photos"], + "review" => ["layout page"], + "publish" => ["review"] +}.freeze + +board = {todo: CARDS.keys.dup, doing: [], done: []} +frames = [] + +move = lambda do |card, from, to| + board[from].delete(card) + board[to] << card + columns = [:todo, :doing, :done].map { |col| board[col] } + height = columns.map(&:size).max + frame = +" %-16s %-16s %-16s\n" % ["TO DO", "DOING", "DONE"] + frame << " #{"-" * 14} #{"-" * 14} #{"-" * 14}\n" + height.times do |i| + frame << (" %-16s %-16s %-16s\n" % columns.map { |col| col[i] || "" }) + end + frames << frame +end + +hooks = { + task_slot_acquired: ->(task_id:, task:, waited:) { move.call(task.description, :todo, :doing) }, + after_task_success: ->(task_id:, task:, result:, duration:) { move.call(task.description, :doing, :done) } +} + +orchestrator = Agentic::PlanOrchestrator.new(concurrency_limit: 2, lifecycle_hooks: hooks) +tasks = {} +CARDS.each do |name, duration| + tasks[name] = Agentic::Task.new( + description: name, + agent_spec: {"name" => name, "instructions" => "do the work"}, + payload: duration + ) + orchestrator.add_task(tasks[name], (DEPS[name] || []).map { |d| tasks.fetch(d) }, + agent: ->(t) { sleep(t.payload) || :done }) +end + +result = orchestrator.execute_plan + +puts "KANBAN (#{frames.size} board states, #{(result.execution_time * 1000).round}ms wall, 2 people on the team)" +puts +# Show the opening, one mid-flight frame, and the final board +[frames.first, frames[frames.size / 2], frames.last].each_with_index do |frame, i| + puts ["at the start:", "mid-flight:", "at the end:"][i] + puts frame +end +puts "every frame above was captured live from lifecycle hooks - the" +puts "board is the plan's actual state, not somebody's memory of it." From b1842cd0a8d7beaaaa3039db2e92509c08f9e062 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 6 Jul 2026 21:24:49 +0000 Subject: [PATCH 04/36] docs: critical path analyzer example (tenderlove round 5) Graph topology + measured durations yields the wall-clock-bounding chain, verified by experiment: an off-path task made instant changes nothing; halving the on-path bottleneck moves the wall clock. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01FeZLdZ3M4q4cho4DZPfSkF --- docs/perspectives/round-5/03-tenderlove.md | 58 +++++++++++++++ examples/critical_path.rb | 86 ++++++++++++++++++++++ 2 files changed, 144 insertions(+) create mode 100644 docs/perspectives/round-5/03-tenderlove.md create mode 100644 examples/critical_path.rb diff --git a/docs/perspectives/round-5/03-tenderlove.md b/docs/perspectives/round-5/03-tenderlove.md new file mode 100644 index 0000000..b218ae1 --- /dev/null +++ b/docs/perspectives/round-5/03-tenderlove.md @@ -0,0 +1,58 @@ +# Round 5 field notes — Aaron Patterson walks the critical path + +*Built: `examples/critical_path.rb` — graph topology plus measured +durations yields the chain that determined the wall clock, then proves +it by experiment.* + +## What I built and why + +The knee finder answered "how many lanes"; this answers the question +that comes right after: "which task is the wall clock's fault?" The +answer is never "all of them" — it's the **critical path**, the longest +duration-weighted chain through the dependency graph: + +``` +wall clock: 341ms +critical path: 340ms = pull:orders -> invoice:month -> report:board +(path explains 100% of the wall clock) +``` + +And because engineers rightly distrust analyzers, the program runs the +experiment instead of asking for faith: make an *off-path* task +instant — wall time unchanged, 341 → 341. Halve the slowest *on-path* +task — 341 → 251. There's your sprint planning: `pull:orders` is the +only work item, and anyone polishing `pull:catalog` is doing charity +for a metric no one measures. + +## What the round-4 accessor made possible + +The path computation is fifteen lines of memoized DFS over +`orchestrator.graph` joined with durations from one hook. In round 3 my +Gantt *rebuilt* the topology by eavesdropping on hooks, which meant it +could only see what executed; the graph accessor knows what was +*declared*, which is what lets the analyzer say "this chain, and no +other chain, bounds you." Sandi's crowbar complaint became my +load-bearing API within one round — that's the compounding this +experiment series keeps demonstrating. + +Matz already filed the follow-up I want: `graph[:order]` (topological). +My DFS re-derives it implicitly; a critical-path tool, a map drawer, +and a scheduler visualizer shouldn't each write their own toposort. + +## Perf-nerd footnotes + +- "Path explains 100% of the wall clock" is itself a diagnostic: if + that number drops much below ~95%, your bottleneck isn't the work, + it's the *scheduler* (queue waits from a too-tight limit). Pair this + with the knee finder: knee for capacity, path for latency. +- The experiment reruns the whole plan with modified sleeps — + affordable here, but the analysis itself is O(V+E) memoized and + needs no rerun. Ship the analysis in CI; save the experiment for + demos and skeptics. + +## Verdict + +Three tools now form a performance suite this framework didn't have +five rounds ago: Gantt (where time went), knee (how many lanes), path +(which task matters). All three built from two hooks and one accessor. +APIs that compound are the ones worth shipping. diff --git a/examples/critical_path.rb b/examples/critical_path.rb new file mode 100644 index 0000000..8d04ca8 --- /dev/null +++ b/examples/critical_path.rb @@ -0,0 +1,86 @@ +# frozen_string_literal: true + +# The Critical Path: after a run, combine the graph topology with +# measured durations to find the chain of tasks that determined the +# wall clock. Optimizing anything OFF that path is charity work - +# and this proves it by making a non-critical task instant and +# re-running. +# +# bundle exec ruby examples/critical_path.rb +# +# Runs offline; durations are simulated IO. + +require_relative "../lib/agentic" + +WORK = { + "pull:catalog" => {sleep: 0.05, deps: []}, + "pull:orders" => {sleep: 0.18, deps: []}, + "pull:reviews" => {sleep: 0.06, deps: []}, + "score:products" => {sleep: 0.09, deps: ["pull:catalog", "pull:reviews"]}, + "invoice:month" => {sleep: 0.12, deps: ["pull:orders"]}, + "report:board" => {sleep: 0.04, deps: ["score:products", "invoice:month"]} +}.freeze + +def run(work, concurrency: 6) + durations = {} + hooks = { + after_task_success: ->(task_id:, task:, result:, duration:) { durations[task_id] = duration } + } + orchestrator = Agentic::PlanOrchestrator.new(concurrency_limit: concurrency, lifecycle_hooks: hooks) + tasks = {} + work.each do |name, spec| + tasks[name] = Agentic::Task.new( + description: name, + agent_spec: {"name" => name, "instructions" => "simulate"}, + payload: spec[:sleep] + ) + orchestrator.add_task(tasks[name], spec[:deps].map { |d| tasks.fetch(d) }, + agent: ->(t) { sleep(t.payload) || :ok }) + end + result = orchestrator.execute_plan + [orchestrator.graph, durations, result.execution_time] +end + +# Longest-duration path to each node, computed over the real topology +def critical_path(graph, durations) + names = graph[:tasks].transform_values(&:description) + memo = {} + walk = lambda do |task_id| + memo[task_id] ||= begin + deps = graph[:dependencies][task_id] + best = deps.map { |dep| walk.call(dep) }.max_by { |p| p[:cost] } || {cost: 0.0, path: []} + {cost: best[:cost] + durations[task_id], path: best[:path] + [names[task_id]]} + end + end + graph[:dependencies].keys.map { |id| walk.call(id) }.max_by { |p| p[:cost] } +end + +graph, durations, wall = run(WORK) +path = critical_path(graph, durations) + +puts "CRITICAL PATH ANALYSIS" +puts +puts format(" wall clock: %dms", wall * 1000) +puts format(" critical path: %dms = %s", path[:cost] * 1000, path[:path].join(" -> ")) +puts format(" (path explains %.0f%% of the wall clock - the rest is scheduling noise)", + 100 * path[:cost] / wall) +puts + +# The proof: optimize a NON-critical task to zero... nothing happens +off_path = WORK.keys.find { |name| !path[:path].include?(name) } +faster = WORK.merge(off_path => WORK[off_path].merge(sleep: 0.0)) +_, _, wall_after_wrong = run(faster) + +# Now halve the SLOWEST task on the path... everything happens +bottleneck = path[:path].max_by { |name| WORK[name][:sleep] } +righter = WORK.merge(bottleneck => WORK[bottleneck].merge(sleep: WORK[bottleneck][:sleep] / 2)) +_, _, wall_after_right = run(righter) + +puts "the experiment:" +puts format(" make '%s' (off-path) instant: %3dms -> %3dms (nothing. told you.)", + off_path, wall * 1000, wall_after_wrong * 1000) +puts format(" halve '%s' (on-path): %3dms -> %3dms (there it is)", + bottleneck, wall * 1000, wall_after_right * 1000) +puts +puts "profile the path, not the plan. optimizing off the critical path" +puts "is how teams burn a sprint making the fast part faster." From da0eea216f819112879df8f9c15aeac6fa53b954 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 6 Jul 2026 21:25:58 +0000 Subject: [PATCH 05/36] docs: Mermaid plan diagrammer example (fxn round 5) orchestrator.graph emitted as Mermaid with named dependencies as labeled edges - documentation generated from the artifact cannot drift from it. Field notes propose graph[:edges] and graph[:order]. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01FeZLdZ3M4q4cho4DZPfSkF --- docs/perspectives/round-5/04-fxn.md | 56 ++++++++++++++++++++++++ examples/plan_diagram.rb | 66 +++++++++++++++++++++++++++++ 2 files changed, 122 insertions(+) create mode 100644 docs/perspectives/round-5/04-fxn.md create mode 100644 examples/plan_diagram.rb diff --git a/docs/perspectives/round-5/04-fxn.md b/docs/perspectives/round-5/04-fxn.md new file mode 100644 index 0000000..b624bc1 --- /dev/null +++ b/docs/perspectives/round-5/04-fxn.md @@ -0,0 +1,56 @@ +# Round 5 field notes — Xavier Noria diagrams the plan + +*Built: `examples/plan_diagram.rb` — `orchestrator.graph` emitted as +Mermaid, with named dependencies as labeled edges.* + +## What I built and why + +Documentation that is generated from the artifact cannot disagree with +the artifact — that is the entire theory of this tool, and it is the +same theory as Zeitwerk's: derive the truth from one source rather +than maintaining two and hoping. `to_mermaid(graph)` is thirty lines; +GitHub renders the output; the diagram in your README regenerates in +CI and can never rot. + +The detail I care most about: **named dependencies become labeled +edges**. `draft` doesn't merely have two arrows into it — the arrows +say `skeleton` and `citations`: + +```mermaid +T1 -- skeleton --> T3 +T2 -- citations --> T3 +``` + +The `needs:` feature was pitched in round 3 as consumer ergonomics +(`t.needs.skeleton` beats positional lookup). This round reveals its +second dividend, which I'd argue is larger: the names are +*architectural documentation that survives extraction*. A diagram that +says WHY an edge exists — not just that it does — is the difference +between a graph and a design. Ergonomics decay into docs; that is the +best career path a parameter name can have. + +## Precision notes + +- The diagrammer must dedupe: `needs:` entries also appear in + `graph[:dependencies]` (correctly — they ARE dependencies), so a + naive emitter draws the edge twice, once bare and once labeled. The + snapshot is honest; consumers must be too. I'd accept a + `graph[:edges]` that pre-merges the two views with labels attached, + as the convenience form. +- Stable node ids (`T0`, `T1`...) derive from insertion order, which + Matz has already noted is a promise nobody has made. Both of us now + want `graph[:order]`. When two cartographers independently request + the same projection, the projection is real. +- I resisted emitting styling, subgraphs, and click handlers. A + generator's output should be a *starting point* that a human can + own, or a terminal artifact that no human touches — the middle + ground (generated-but-hand-tweaked) is where diagrams go to rot, + Mermaid or not. + +## Verdict + +The graph accessor is one round old and has now fed a game map, a +critical-path analyzer, a design critic, and a documentation +generator. Expose the right projection and an ecosystem assembles +itself around it — loaders taught me that; this gem is re-learning it +in public, quickly. diff --git a/examples/plan_diagram.rb b/examples/plan_diagram.rb new file mode 100644 index 0000000..7ea87d5 --- /dev/null +++ b/examples/plan_diagram.rb @@ -0,0 +1,66 @@ +# frozen_string_literal: true + +# The Plan Diagrammer: any orchestrator's graph, emitted as Mermaid - +# paste it into a README, GitHub renders it, and the diagram can never +# drift from the plan because it is generated FROM the plan. +# +# bundle exec ruby examples/plan_diagram.rb +# +# Runs offline; prints Mermaid source. Named dependencies become +# labeled edges; plain dependencies become arrows. + +require_relative "../lib/agentic" + +# A representative plan: the editorial pipeline with a named fan-in +orchestrator = Agentic::PlanOrchestrator.new + +def step(name) + Agentic::Task.new(description: name, agent_spec: {"name" => name, "instructions" => "work"}) +end + +research = step("research topic") +outline = step("draft outline") +sources = step("verify sources") +draft = step("write draft") +publish = step("publish") + +orchestrator.add_task(research) +orchestrator.add_task(outline, [research]) +orchestrator.add_task(sources, [research]) +orchestrator.add_task(draft, needs: {skeleton: outline, citations: sources}) +orchestrator.add_task(publish, [draft]) + +# --- the diagrammer: graph in, mermaid out ----------------------------------- +def to_mermaid(graph) + names = graph[:tasks].transform_values(&:description) + ids = names.keys.each_with_index.to_h { |task_id, i| [task_id, "T#{i}"] } + + named_edges = graph[:needs].flat_map { |task_id, needs| + needs.map { |label, dep_id| [dep_id, task_id, label] } + } + labeled = named_edges.to_h { |from, to, _| [[from, to], true] } + + lines = ["graph TD"] + names.each { |task_id, name| lines << " #{ids[task_id]}[\"#{name}\"]" } + graph[:dependencies].each do |task_id, deps| + deps.each do |dep_id| + next if labeled[[dep_id, task_id]] # named edges are drawn with labels below + + lines << " #{ids[dep_id]} --> #{ids[task_id]}" + end + end + named_edges.each do |from, to, label| + lines << " #{ids[from]} -- #{label} --> #{ids[to]}" + end + lines.join("\n") +end + +mermaid = to_mermaid(orchestrator.graph) + +puts "```mermaid" +puts mermaid +puts "```" +puts +puts "paste the block above into any GitHub markdown file. the arrows" +puts "labeled 'skeleton' and 'citations' are the named dependencies -" +puts "the diagram documents not just THAT draft waits, but WHY." From 090cb2462b486ef7a97e429e61397a5a28d64cab Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 6 Jul 2026 21:27:04 +0000 Subject: [PATCH 06/36] docs: burst absorber example (ioquatix round 5) Characterizes Agentic::RateLimit under three request waves: the ceiling holds, the queue absorbs, and the per-wave wait table is the capacity plan. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01FeZLdZ3M4q4cho4DZPfSkF --- docs/perspectives/round-5/05-ioquatix.md | 57 ++++++++++++++++++++++++ examples/burst_absorber.rb | 55 +++++++++++++++++++++++ 2 files changed, 112 insertions(+) create mode 100644 docs/perspectives/round-5/05-ioquatix.md create mode 100644 examples/burst_absorber.rb diff --git a/docs/perspectives/round-5/05-ioquatix.md b/docs/perspectives/round-5/05-ioquatix.md new file mode 100644 index 0000000..782eff4 --- /dev/null +++ b/docs/perspectives/round-5/05-ioquatix.md @@ -0,0 +1,57 @@ +# Round 5 field notes — Samuel Williams absorbs the burst + +*Built: `examples/burst_absorber.rb` — three request waves against +`Agentic::RateLimit` (this round's release); the table of wait times +is the capacity plan.* + +## What I built and why + +Round 4 I built the rate limiter from primitives and said "this is one +class away from a feature." The class shipped (`Agentic::RateLimit`, +plus `LlmClient limiter:`), so this round I characterized it the way +you'd characterize any queueing system — under bursts, because steady +state flatters everyone: + +``` +wave 1: 6 requests wait p50 50ms worst 50ms +wave 2: 2 requests wait p50 0ms worst 0ms +wave 3: 9 at once wait p50 50ms worst 100ms +high-water mark: 3 of 3 - held +``` + +Six into three slots: one 50ms queueing round. Two requests: through +untouched. Nine at once: the tail waits two full service times. That +last number is the one to frame — **a ceiling doesn't eliminate burst +cost, it converts it** from a provider-side 429 (opaque, retried, +billed twice) into local queueing (visible, measurable, bounded). The +wait table IS the capacity conversation: if 100ms of p-worst is +unacceptable at wave-3 volume, you need a second credential, and now +you know *before* production tells you. + +## Design review of my own feature + +- The `ensure` inside `RateLimit#acquire` — decrementing in-flight even + when the block raises — is the difference between a limiter and a + slow leak. I checked the shipped implementation first thing; it's + there. Review your own requests hardest. +- High-water as a first-class reader means every demo doubles as a + regression test ("3 of 3 - held"). Observability designed into the + primitive, not bolted on. +- What it doesn't do (correctly, for now): time-windowed limits + (requests-per-minute vs concurrent). Concurrency ceilings model + connection limits; token buckets model quota. Both belong eventually, + but shipping the semaphore-shaped one first was right — it's the one + that can't be approximated from outside. + +## Structured-concurrency footnote + +The waves themselves are nested `task.async` fan-outs with `sleep` +staggering — the test harness for the feature is the same three +primitives as the feature. When your testing DSL is just your +concurrency model, the concurrency model is probably right. + +## Verdict + +Feature requested, shipped, and characterized under hostile load +within two rounds. The wait table converts "we should rate limit" from +compliance theater into an engineering table with numbers in it. diff --git a/examples/burst_absorber.rb b/examples/burst_absorber.rb new file mode 100644 index 0000000..f7d98f9 --- /dev/null +++ b/examples/burst_absorber.rb @@ -0,0 +1,55 @@ +# frozen_string_literal: true + +# The Burst Absorber: three waves of requests slam a credential with a +# ceiling of 3 (Agentic::RateLimit - this round's release). The ceiling +# holds, the queue absorbs, and the per-request wait times show exactly +# what "absorbed" costs. Capacity planning is reading this table. +# +# bundle exec ruby examples/burst_absorber.rb +# +# Runs offline; the upstream is sleep. + +require_relative "../lib/agentic" +require "async" + +CEILING = 3 +CALL_TIME = 0.05 +WAVES = [6, 2, 9].freeze # requests arriving together, 120ms apart + +limit = Agentic::RateLimit.new(CEILING) +waits = Hash.new { |h, k| h[k] = [] } + +puts "BURST ABSORBER: ceiling #{CEILING}, calls take #{(CALL_TIME * 1000).round}ms" +puts + +Sync do |task| + arrivals = [] + WAVES.each_with_index do |count, wave| + arrivals << task.async do + sleep(wave * 0.12) # the wave arrives + count.times.map { |i| + task.async do + arrived = Process.clock_gettime(Process::CLOCK_MONOTONIC) + limit.acquire do + waits[wave] << Process.clock_gettime(Process::CLOCK_MONOTONIC) - arrived + sleep(CALL_TIME) + end + end + }.each(&:wait) + end + end + arrivals.each(&:wait) +end + +WAVES.each_with_index do |count, wave| + wave_waits = waits[wave].sort + puts format(" wave %d: %d requests wait p50 %3dms worst %3dms", + wave + 1, count, wave_waits[wave_waits.size / 2] * 1000, wave_waits.last * 1000) +end + +puts +puts format(" high-water mark: %d of %d - the ceiling held through every burst", limit.high_water, CEILING) +puts +puts "wave 1 (6 into 3 slots) queues; wave 2 (2 requests) sails through;" +puts "wave 3 (9 at once) pays the real price. the ceiling converts" +puts "provider 429s into local queueing - visible, measurable, bounded." From 3dfbe98c048308abb6c3a039bb2c360b12c3c7e9 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 6 Jul 2026 21:28:13 +0000 Subject: [PATCH 07/36] docs: freight desk example (Jeremy Evans round 5) A tariff book as cross-field contract rules: all broken rules reported at once, per-key validation shielding the rules from nonsense. Policy and enforcement as one artifact. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01FeZLdZ3M4q4cho4DZPfSkF --- docs/perspectives/round-5/06-jeremyevans.md | 57 ++++++++++++++++ examples/freight_rules.rb | 72 +++++++++++++++++++++ 2 files changed, 129 insertions(+) create mode 100644 docs/perspectives/round-5/06-jeremyevans.md create mode 100644 examples/freight_rules.rb diff --git a/docs/perspectives/round-5/06-jeremyevans.md b/docs/perspectives/round-5/06-jeremyevans.md new file mode 100644 index 0000000..48d6a0b --- /dev/null +++ b/docs/perspectives/round-5/06-jeremyevans.md @@ -0,0 +1,57 @@ +# Round 5 field notes — Jeremy Evans opens the freight desk + +*Built: `examples/freight_rules.rb` — a quoting capability whose +tariff policy is written as cross-field contract rules; broken rules +are reported all at once, nonsense never reaches them.* + +## What I built and why + +The cross-field rules Piotr and I asked for shipped this round, so I +built the workload that motivated the ask: a tariff book. Freight +policy is inherently cross-field — "hazardous cargo may not fly" is a +relationship between `mode` and `hazardous`; no per-key check can say +it. Four rules on the contract, five manifests through the desk: + +``` +#3 REFUSED - 3 rule(s) broken: + - air freight is limited to 500kg + - hazardous cargo may not fly + - insured value over 100k requires sea mode +#5 MALFORMED - mode, weight_kg, destination invalid + (never reached the tariff book) +``` + +Two properties I care about, both verified: + +1. **All broken rules at once.** Manifest #3 violates three policies + and hears about all three. First-failure reporting turns a manifest + correction into a submit-fail-submit-fail scavenger hunt — three + round trips where one suffices. The implementation runs every rule + and collects; that's the only correct behavior. +2. **Layering.** Manifest #5 (`mode: "teleport"`, negative weight, + empty destination) is rejected by per-key validation and the rules + never run. This is a *soundness* requirement disguised as + politeness: rule predicates dereference fields + (`i[:weight_kg] <= 500`), and running them against garbage yields + NoMethodErrors or — worse — accidental passes. Types first, then + relationships. The validator sequences it correctly. + +## What I'd note for round 6 + +- Rules are keyed by their description string, which doubles as the + error message. Good for humans; for machine consumers (Piotr's form + renderer next door) a structured `{rule: :air_weight_limit, + message: "..."}` pair would version better. Strings are UI; symbols + are API. +- My round-3 fuzzer can't fuzz rules — a lambda is opaque to input + generation. That's an acceptable trade (expressiveness beat + analyzability here, correctly), but it means rule-bearing contracts + need hand-written property tests. The freight desk's five manifests + are that, in miniature. + +## Verdict + +Policy as data on the contract, checked in the right order, reported +in full. The tariff book and the validation are now the same artifact +— which means the policy can't drift from its enforcement. That's the +property every compliance system claims and almost none have. diff --git a/examples/freight_rules.rb b/examples/freight_rules.rb new file mode 100644 index 0000000..605f96a --- /dev/null +++ b/examples/freight_rules.rb @@ -0,0 +1,72 @@ +# frozen_string_literal: true + +# The Freight Desk: a quoting capability whose tariff book is written +# as cross-field contract rules (new this round). Per-key checks catch +# nonsense; rules: catch the LEGAL-LOOKING orders that violate policy - +# and every broken rule is reported at once, because a shipper fixing +# their manifest deserves the whole list, not a scavenger hunt. +# +# bundle exec ruby examples/freight_rules.rb +# +# Runs offline and deterministically. + +require_relative "../lib/agentic" + +spec = Agentic::CapabilitySpecification.new( + name: "quote_freight", + description: "Quote a freight shipment", + version: "1.0.0", + inputs: { + mode: {type: "string", required: true, enum: %w[air sea road]}, + weight_kg: {type: "number", required: true, min: 1, max: 30_000}, + hazardous: {type: "boolean", required: true}, + insured_value: {type: "number", required: true, min: 0}, + destination: {type: "string", required: true, non_empty: true} + }, + rules: { + "air freight is limited to 500kg" => + ->(i) { i[:mode] != "air" || i[:weight_kg] <= 500 }, + "hazardous cargo may not fly" => + ->(i) { !(i[:mode] == "air" && i[:hazardous]) }, + "insured value over 100k requires sea mode" => + ->(i) { i[:insured_value] <= 100_000 || i[:mode] == "sea" }, + "road freight only reaches domestic destinations" => + ->(i) { i[:mode] != "road" || i[:destination].start_with?("domestic:") } + } +) + +RATES = {"air" => 4.20, "sea" => 0.30, "road" => 1.10}.freeze +Agentic.register_capability(spec, Agentic::CapabilityProvider.new( + capability: spec, + implementation: ->(i) { {quote: (i[:weight_kg] * RATES.fetch(i[:mode])).round(2)} } +)) + +desk = Agentic::AgentCapabilityRegistry.instance.get_provider("quote_freight") + +MANIFESTS = [ + {mode: "sea", weight_kg: 12_000, hazardous: true, insured_value: 250_000, destination: "port of rotterdam"}, + {mode: "air", weight_kg: 480, hazardous: false, insured_value: 20_000, destination: "berlin"}, + {mode: "air", weight_kg: 900, hazardous: true, insured_value: 150_000, destination: "tokyo"}, + {mode: "road", weight_kg: 2_000, hazardous: false, insured_value: 5_000, destination: "domestic:austin"}, + {mode: "teleport", weight_kg: -5, hazardous: false, insured_value: 10, destination: ""} +].freeze + +puts "THE FREIGHT DESK (#{spec.rules.size} tariff rules on the contract)" +puts +MANIFESTS.each_with_index do |manifest, index| + quote = desk.execute(manifest) + puts format(" #%d QUOTED $%.2f (%s, %dkg)", index + 1, quote[:quote], manifest[:mode], manifest[:weight_kg]) +rescue Agentic::Errors::ValidationError => e + if e.violations.key?(:base) + puts " ##{index + 1} REFUSED - #{e.violations[:base].size} rule(s) broken:" + e.violations[:base].each { |rule| puts " - #{rule}" } + else + puts " ##{index + 1} MALFORMED - #{e.violations.keys.join(", ")} invalid " \ + "(never reached the tariff book)" + end +end + +puts +puts "manifest #3 broke THREE rules and heard about all three at once." +puts "manifest #5 never reached the tariff book: per-key validation" +puts "rejects nonsense before cross-field rules spend time on it." From 88c5190829355fad6bb21c1f948af45ef69c4ef9 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 6 Jul 2026 21:29:16 +0000 Subject: [PATCH 08/36] docs: 422 generator example (solnic round 5) A contract-agnostic error renderer built entirely from ValidationError#expectations - allowed values and bounds travel inside the exception, so one renderer serves every capability. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01FeZLdZ3M4q4cho4DZPfSkF --- docs/perspectives/round-5/07-solnic.md | 55 ++++++++++++++++++ examples/form_errors.rb | 78 ++++++++++++++++++++++++++ 2 files changed, 133 insertions(+) create mode 100644 docs/perspectives/round-5/07-solnic.md create mode 100644 examples/form_errors.rb diff --git a/docs/perspectives/round-5/07-solnic.md b/docs/perspectives/round-5/07-solnic.md new file mode 100644 index 0000000..68ff6c9 --- /dev/null +++ b/docs/perspectives/round-5/07-solnic.md @@ -0,0 +1,55 @@ +# Round 5 field notes — Piotr Solnica generates the 422 + +*Built: `examples/form_errors.rb` — ValidationError in, API error +document out; the renderer has zero knowledge of any contract because +`#expectations` (this round's release) carries the contract inside the +exception.* + +## What I built and why + +My round-4 state machine had to reach back into its own transition +table to tell users which states were legal — the exception knew +something was wrong but not what right looked like. `#expectations` +closed that gap, and this is the payoff pattern: **one renderer, every +capability**. The `error_document` function receives an exception and +produces the 422 your frontend wants: + +```json +{"field": "plan", + "messages": ["must be one of: starter, team, enterprise"], + "allowed": ["starter", "team", "enterprise"]} +``` + +`allowed`, `minimum`, `maximum` — all read off the exception. The +renderer contains no reference to the checkout contract, which means +it also serves the freight desk next door, and every capability anyone +registers next year. Error rendering just became infrastructure +instead of per-endpoint labor. This is the dry-rb thesis in one +example: when errors are *data with schema*, presentation becomes a +pure function, and pure functions are the only code that never needs +a second copy. + +## The seam I'd polish next + +The document splits field errors from `policy_violations` (the `:base` +rules) — and submission #3 shows the asymmetry Jeremy also flagged: +field errors carry structured expectations, policy violations carry +prose. A rule that failed *knows* which fields it read +(`plan`, `seats`), but the exception can't say so, so the frontend +can't highlight the offending inputs. Rules declaring their fields — +`rules: {rule_name => {fields: [:plan, :seats], check: ->}}` — would +let the 422 point at widgets for policy failures too. That's the +round-6 ask. + +Also noted with approval: dry-schema's own messages ("must be one of: +starter, team, enterprise") already render the enum — the expectations +payload lets you go *beyond* prose into machine-usable structure +(populate the dropdown, set the slider bounds). Messages for humans, +expectations for widgets. Both, not either. + +## Verdict + +Round 4: exceptions learned what happened. Round 5: they learned what +should have happened. A form library, an API layer, and a CLI can now +share one error pathway — which is what "types at the boundary" was +always for. diff --git a/examples/form_errors.rb b/examples/form_errors.rb new file mode 100644 index 0000000..0d601c1 --- /dev/null +++ b/examples/form_errors.rb @@ -0,0 +1,78 @@ +# frozen_string_literal: true + +# The 422 Generator: turn a ValidationError into the API error document +# your frontend actually wants - message, allowed values, bounds - using +# ONLY what the exception carries (new this round: #expectations). The +# renderer has zero knowledge of the contract; the exception brings the +# contract with it. +# +# bundle exec ruby examples/form_errors.rb +# +# Runs offline; prints the JSON your form would receive. + +require_relative "../lib/agentic" +require "json" + +spec = Agentic::CapabilitySpecification.new( + name: "checkout", + description: "Process a checkout form", + version: "1.0.0", + inputs: { + email: {type: "string", required: true, non_empty: true}, + plan: {type: "string", required: true, enum: %w[starter team enterprise]}, + seats: {type: "number", required: true, min: 1, max: 500}, + coupon: {type: "string"} + }, + rules: { + "starter plan is limited to 5 seats" => ->(i) { i[:plan] != "starter" || i[:seats] <= 5 } + } +) +Agentic.register_capability(spec, Agentic::CapabilityProvider.new( + capability: spec, implementation: ->(i) { {order_id: "ord-#{i[:plan]}-#{i[:seats]}"} } +)) + +# The renderer: exception in, error document out. Note what it does NOT +# have: any reference to the checkout contract. +def error_document(error) + field_errors = error.violations.filter_map do |field, messages| + next if field == :base + + declared = error.expectations[field] || {} + detail = {field: field, messages: Array(messages)} + detail[:allowed] = declared[:enum] if declared[:enum] + detail[:minimum] = declared[:min] if declared[:min] + detail[:maximum] = declared[:max] if declared[:max] + detail[:type] = declared[:type] if declared[:type] + detail + end + + { + status: 422, + capability: error.capability, + errors: field_errors, + policy_violations: error.violations.fetch(:base, []) + } +end + +checkout = Agentic::AgentCapabilityRegistry.instance.get_provider("checkout") + +SUBMISSIONS = [ + {email: "ada@example.com", plan: "team", seats: 12}, + {email: "", plan: "premium", seats: 0}, + {email: "joan@example.com", plan: "starter", seats: 9} +].freeze + +SUBMISSIONS.each_with_index do |form, index| + puts "submission ##{index + 1}: #{form.inspect}" + begin + result = checkout.execute(form) + puts " 201 CREATED #{result[:order_id]}" + rescue Agentic::Errors::ValidationError => e + puts JSON.pretty_generate(error_document(e)).gsub(/^/, " ") + end + puts +end + +puts "the renderer never saw the checkout contract - 'allowed', 'minimum'," +puts "and 'maximum' all traveled inside the exception. one renderer serves" +puts "every capability in the app, current and future." From 2077bbb4b8600f6ba43c5ad502f5a6b32ee2482b Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 6 Jul 2026 21:30:44 +0000 Subject: [PATCH 09/36] docs: stampede simulator example (mperham round 5) Twenty synchronized retries versus the new jitter-on default, as a seeded, reproducible histogram: peak herd 20 -> 11. Field notes file the full-jitter tier as the next notch. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01FeZLdZ3M4q4cho4DZPfSkF --- docs/perspectives/round-5/08-mperham.md | 49 ++++++++++++++ examples/stampede_sim.rb | 87 +++++++++++++++++++++++++ 2 files changed, 136 insertions(+) create mode 100644 docs/perspectives/round-5/08-mperham.md create mode 100644 examples/stampede_sim.rb diff --git a/docs/perspectives/round-5/08-mperham.md b/docs/perspectives/round-5/08-mperham.md new file mode 100644 index 0000000..c0c5bed --- /dev/null +++ b/docs/perspectives/round-5/08-mperham.md @@ -0,0 +1,49 @@ +# Round 5 field notes — Mike Perham simulates the stampede + +*Built: `examples/stampede_sim.rb` — twenty workers fail together and +retry; the histogram compares the herd with jitter off versus the +now-default jitter on.* + +## What I built and why + +I've been muttering "jitter by default" since round 3; it shipped this +round, and defaults this important deserve a demonstration you can +show a skeptical platform team. Twenty workers all fail at t=0 (an +upstream hiccup — the most common failure shape there is, because +outages are *correlated*), all back off 120ms, all return: + +``` +jitter OFF: 220-240ms #################### 20 +jitter ON: 80-100ms # 1 + 100-120ms ########### 11 + 120-140ms #### 4 + 140-160ms #### 4 +peak herd: 20 -> 11 +``` + +Without jitter, the recovery *is* the second outage: twenty requests +in one 20ms bucket, aimed at an upstream that just told you it was +struggling. With ±25% jitter the same herd spreads across 80ms and the +peak nearly halves. At twenty workers this is a chart; at two thousand +it's the difference between recovery and a cascading failure with your +company's name on the postmortem. + +## Honest notes on the experiment + +- Both runs are seeded (`srand`) so the comparison is fair and the + histogram is reproducible — a stampede simulator with + unreproducible stampedes is weather, not science. +- Peak 11 of 20 still isn't great — uniform ±25% jitter caps how much + spread you can buy. The literature's answer is *full jitter* + (`rand(0..delay)`) or decorrelated jitter, which flatten the herd + much harder. The framework's knob is a boolean today; a + `backoff_jitter: :full` tier is the natural next notch. Filed. +- The example silences the logger to `:fatal` because forty scripted + failures are the point, not news — and being *able* to do that in + one line is Jeremy's round-1 logger work paying rent again. + +## Verdict + +The default changed, and now there's a picture explaining why nobody +should change it back. Reliability defaults should assume the crowd; +this one finally does. diff --git a/examples/stampede_sim.rb b/examples/stampede_sim.rb new file mode 100644 index 0000000..9e388c9 --- /dev/null +++ b/examples/stampede_sim.rb @@ -0,0 +1,87 @@ +# frozen_string_literal: true + +# The Stampede Simulator: twenty workers hit a hiccuping upstream, all +# fail at once, all retry. With jitter OFF they come back as a single +# synchronized herd - the second outage. With jitter ON (the default, +# as of this round) the herd spreads. The histogram is the argument. +# +# bundle exec ruby examples/stampede_sim.rb [seed] +# +# Runs offline; the upstream is a shared counter with feelings. + +require_relative "../lib/agentic" + +# 40 scripted failures are the point, not news +Agentic.logger.level = :fatal + +WORKERS = 20 +BACKOFF = 0.12 +BUCKET_MS = 20 + +def run_stampede(jitter:, seed:) + srand(seed) # jitter uses Kernel#rand; seed it for a fair comparison + started = Process.clock_gettime(Process::CLOCK_MONOTONIC) + retry_arrivals = [] + attempts = Hash.new(0) + + orchestrator = Agentic::PlanOrchestrator.new( + concurrency_limit: WORKERS, + retry_policy: { + max_retries: 2, + retryable_errors: ["RuntimeError"], + backoff_strategy: :constant, + backoff_constant: BACKOFF, + backoff_jitter: jitter + } + ) + + WORKERS.times do |i| + task = Agentic::Task.new( + description: "worker-#{i}", + agent_spec: {"name" => "worker", "instructions" => "call upstream"} + ) + orchestrator.add_task(task, agent: ->(t) { + attempts[t.description] += 1 + if attempts[t.description] == 1 + raise "upstream hiccup" # everyone fails together at t=0 + end + + retry_arrivals << Process.clock_gettime(Process::CLOCK_MONOTONIC) - started + :ok + }) + end + + orchestrator.execute_plan + retry_arrivals +end + +def histogram(arrivals) + buckets = Hash.new(0) + arrivals.each { |t| buckets[(t * 1000 / BUCKET_MS).floor * BUCKET_MS] += 1 } + buckets.sort.map { |bucket_ms, count| + format(" %4d-%4dms %-20s %d", bucket_ms, bucket_ms + BUCKET_MS, "#" * count, count) + }.join("\n") +end + +seed = (ARGV.first || 20260707).to_i + +puts "STAMPEDE SIMULATOR: #{WORKERS} workers, all failing at t=0, " \ + "#{(BACKOFF * 1000).round}ms constant backoff" +puts + +herd = run_stampede(jitter: false, seed: seed) +puts " jitter OFF (retry arrivals per #{BUCKET_MS}ms bucket):" +puts histogram(herd) +peak_off = herd.group_by { |t| (t * 1000 / BUCKET_MS).floor }.values.map(&:size).max + +spread = run_stampede(jitter: true, seed: seed) +puts +puts " jitter ON - the default (same workers, same failure, same seed):" +puts histogram(spread) +peak_on = spread.group_by { |t| (t * 1000 / BUCKET_MS).floor }.values.map(&:size).max + +puts +puts format(" peak herd size per bucket: %d without jitter -> %d with", peak_off, peak_on) +puts +puts "twenty synchronized retries is a second outage wearing a recovery" +puts "costume. jitter is on by default now; the upstream sends its thanks." From 89c96c58777b88710abef8b6ffbb69b316ca0687 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 6 Jul 2026 21:31:52 +0000 Subject: [PATCH 10/36] docs: three shapes example (Sandi Metz round 5) One workload as chain, star, and staged joins - wall time measured, depth and fan-in read from the graph, trades stated. Star and staged tie on speed, making the choice purely about design. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01FeZLdZ3M4q4cho4DZPfSkF --- docs/perspectives/round-5/09-sandimetz.md | 56 +++++++++++++++++ examples/three_shapes.rb | 73 +++++++++++++++++++++++ 2 files changed, 129 insertions(+) create mode 100644 docs/perspectives/round-5/09-sandimetz.md create mode 100644 examples/three_shapes.rb diff --git a/docs/perspectives/round-5/09-sandimetz.md b/docs/perspectives/round-5/09-sandimetz.md new file mode 100644 index 0000000..7175a82 --- /dev/null +++ b/docs/perspectives/round-5/09-sandimetz.md @@ -0,0 +1,56 @@ +# Round 5 field notes — Sandi Metz teaches the three shapes + +*Built: `examples/three_shapes.rb` — the same six units of work as a +chain, a star, and staged joins; measured, critiqued, and chosen on +purpose.* + +## What I built and why + +My graph critic (round 4) could smell a bad shape; this round's tool +teaches the *choosing*. Same workload, three arrangements, and for +each: wall time (measured by running it) plus two structural numbers +read straight off `orchestrator.graph` — depth and max fan-in. One +table, three trades: + +``` +chain 243ms depth 6 fan-in 1 trivially debuggable; pays serial price +star 121ms depth 3 fan-in 4 fastest; one join owns every failure mode +staged 121ms depth 3 fan-in 3 nearly as fast; each join has one reason to wait +``` + +The lesson I've taught for years about objects transfers whole: **none +of these is wrong.** The chain is right when the work is truly +sequential. The star is right when the join is trivial. The staged +shape is right when the join has judgment in it, because judgment +deserves small, single-purpose homes. What's *wrong* is not knowing +which one you built — accidental architecture, the graph edition. + +Note what the measurement adds that intuition misses: star and staged +tie on wall time (121ms — the scheduler forgives fan-in that fits the +concurrency limit). So the choice between them is **entirely** about +failure modes and change cost, not speed — which is exactly the +conversation teams skip when they only look at latency. The numbers +free you to argue about design. + +## On building teaching tools with this framework + +- The structural facts are eight lines over the graph accessor; the + behavioral fact is one `execute_plan`. Structure cheap, behavior + honest — a teaching tool needs both, because students (rightly) + distrust tools that only theorize. +- This is my third graph tool in three rounds (critic, tracer, + shapes), and they compose into a curriculum: *see* the conversation + (tracer), *smell* the structure (critic), *choose* the shape + (shapes). A workshop hiding in an examples directory. +- The depth/fan-in computation is now written in three examples + (critic, critical path, here). When teaching materials repeat a + computation, the library wants it: `graph[:depth]`, `graph[:fan_in]` + — or Matz and Xavier's `graph[:order]` plus a tiny stats helper. + Third strike; extract it. + +## Verdict + +Design education in forty lines: run the shapes, read the table, have +the argument the numbers permit. When a framework makes trade-offs +this cheap to *demonstrate*, "it depends" stops being a shrug and +becomes a lesson plan. diff --git a/examples/three_shapes.rb b/examples/three_shapes.rb new file mode 100644 index 0000000..0be7b06 --- /dev/null +++ b/examples/three_shapes.rb @@ -0,0 +1,73 @@ +# frozen_string_literal: true + +# Three Shapes: the same six units of work arranged three ways - a +# chain, a star, and staged joins - then measured and critiqued. Design +# is choosing a shape ON PURPOSE, and purpose needs numbers. +# +# bundle exec ruby examples/three_shapes.rb +# +# Runs offline; each unit of work is 40ms of simulated IO. + +require_relative "../lib/agentic" + +UNIT = 0.04 +NAMES = %w[gather_a gather_b gather_c gather_d combine finish].freeze + +def build(shape) + orchestrator = Agentic::PlanOrchestrator.new(concurrency_limit: 4) + tasks = NAMES.to_h do |name| + [name, Agentic::Task.new(description: name, agent_spec: {"name" => name, "instructions" => "work"})] + end + + deps = case shape + when :chain # one long line: simple to read, nothing overlaps + NAMES.each_cons(2).to_h { |a, b| [b, [a]] } + when :star # everything at once, one god join + {"combine" => %w[gather_a gather_b gather_c gather_d], "finish" => ["combine"]} + when :staged # balanced: pairs join, then join the joins + {"gather_c" => [], "gather_d" => [], + "combine" => %w[gather_a gather_b], + "finish" => %w[combine gather_c gather_d]} + end + + NAMES.each do |name| + orchestrator.add_task(tasks[name], (deps[name] || []).map { |d| tasks.fetch(d) }, + agent: ->(_t) { sleep(UNIT) || :ok }) + end + orchestrator +end + +# Structural facts, read straight off the graph +def shape_facts(graph) + dependencies = graph[:dependencies] + depth = {} + measure = ->(id) { depth[id] ||= 1 + (dependencies[id].map { |d| measure.call(d) }.max || 0) } + { + max_fan_in: dependencies.values.map(&:size).max, + depth: dependencies.keys.map { |id| measure.call(id) }.max + } +end + +puts "THREE SHAPES: six tasks x #{(UNIT * 1000).round}ms, concurrency 4" +puts +puts format(" %-8s %-10s %-8s %-9s %s", "shape", "wall", "depth", "max fan-in", "the trade") + +TRADES = { + chain: "trivially debuggable; pays full serial price", + star: "fastest; one join owns every failure mode", + staged: "nearly as fast; each join has one reason to wait" +}.freeze + +%i[chain star staged].each do |shape| + orchestrator = build(shape) + facts = shape_facts(orchestrator.graph) + result = orchestrator.execute_plan + puts format(" %-8s %4dms %-8d %-9d %s", + shape, result.execution_time * 1000, facts[:depth], facts[:max_fan_in], TRADES[shape]) +end + +puts +puts "none of these is wrong. the chain is right when the work is truly" +puts "sequential; the star when the join is trivial; the staged shape when" +puts "the join has judgment in it. what's wrong is not knowing which one" +puts "you built - and now the graph will tell you, in two numbers." From 796c84a4673ca02bdd69d1acb9882a6c04773db3 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 6 Jul 2026 21:33:31 +0000 Subject: [PATCH 11/36] docs: examples index generator (ankane round 5) Generates examples/README.md from each example's own header and fails on any example missing a run line - forty examples cataloged, self-maintaining signage for the gallery. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01FeZLdZ3M4q4cho4DZPfSkF --- docs/perspectives/round-5/10-ankane.md | 52 +++++++++++++++++ examples/README.md | 48 ++++++++++++++++ examples/examples_index.rb | 77 ++++++++++++++++++++++++++ 3 files changed, 177 insertions(+) create mode 100644 docs/perspectives/round-5/10-ankane.md create mode 100644 examples/README.md create mode 100644 examples/examples_index.rb diff --git a/docs/perspectives/round-5/10-ankane.md b/docs/perspectives/round-5/10-ankane.md new file mode 100644 index 0000000..f1dbe64 --- /dev/null +++ b/docs/perspectives/round-5/10-ankane.md @@ -0,0 +1,52 @@ +# Round 5 field notes — Andrew Kane catalogs the gallery + +*Built: `examples/examples_index.rb` — reads every example's own +header, writes `examples/README.md`, and fails loudly on any example +that doesn't document how to run itself.* + +## What I built and why + +Five rounds produced forty examples — a gallery with no signage. The +signage problem has the same solution as the README-rot problem I +attacked in round 4: **generate the catalog from the artifacts**, so +it can't disagree with them. The librarian tasks fan out (one per +file), each reads its example's leading comment block, and the editor +fans in and writes the table. `examples/README.md` now exists, says +"edit the examples, not this file," and regenerates in one command. + +The enforcement matters as much as the catalog: an example whose +header lacks a `bundle exec ruby` line fails the build (exit 1), +because **an example you can't run is a rumor**. Forty of forty pass +today — every persona, it turns out, wrote honest headers. Culture +audits itself when tools check it. + +## Confession, as gallery tradition requires + +First run cataloged all forty summaries as +"frozen_string_literal: true" — my header extractor grabbed the first +comment it saw, and the first comment in every file is the magic +comment. Parsing "the comment block" means knowing which comments are +prose and which are pragmas; even a forty-line tool meets the +regexes-vs-grammar lesson eventually. (Aaron is legally entitled to +one laugh.) + +## Patterns worth naming, five rounds in + +- This is the survey/atlas shape's **seventh** appearance, and its + most meta: the framework cataloging the demonstrations of itself. +- The generated file is committed, not gitignored — same call as + Xavier's Mermaid: a catalog you can read on GitHub beats a catalog + you must generate to see, and CI regenerating + diffing keeps it + honest (`examples_index.rb && git diff --exit-code examples/README.md` + is the whole check). +- Self-excluding (`files - [me]`) keeps the librarian out of her own + catalog. Tools that inventory a directory they live in always need + this line, and always forget it once. + +## Verdict + +The gallery has signage that maintains itself and a doorman that +rejects undocumented exhibits. Between the README verifier, the +contract fuzzer, and this index, the project's honesty checks now +cover code promises, contract promises, and catalog promises — all +generated, all CI-able, all built on one gem's primitives. diff --git a/examples/README.md b/examples/README.md new file mode 100644 index 0000000..597b1a0 --- /dev/null +++ b/examples/README.md @@ -0,0 +1,48 @@ +# Examples + +Every example runs offline: `bundle exec ruby examples/`. +This index is generated by `examples_index.rb`; edit the examples, +not this file. + +| Example | What it shows | +|---------|---------------| +| `burst_absorber.rb` | The Burst Absorber: three waves of requests slam a credential with a ceiling of 3 (Agentic::RateLimit - this round's rel... | +| `changelog_scout.rb` | The Changelog Scout: reads real git history, classifies every commit through a contract-checked capability, and drafts t... | +| `collaboration_tracer.rb` | The Collaboration Tracer: lifecycle hooks record every message the orchestrator sends and every reply that comes back, t... | +| `command_bus.rb` | The Command Bus: every command is a composed capability with its OWN declared contract (new in this round - compositions... | +| `contract_fuzzer.rb` | The Contract Fuzzer: for every registered capability, generate inputs that SHOULD pass its declared contract and mutatio... | +| `coupling_cartographer.rb` | The Coupling Cartographer: which files lean on which? Every file is surveyed for the constants it DEFINES and the consta... | +| `critical_path.rb` | The Critical Path: after a run, combine the graph topology with measured durations to find the chain of tasks that deter... | +| `doc_coverage.rb` | The Documentation Surveyor: measures YARD comment coverage for every public method in a lib/ tree. One survey task per f... | +| `dungeon_crawl.rb` | The Dungeon Crawl: a quest is a plan, rooms are tasks, and doors are dependencies. The map is drawn from the orchestrato... | +| `durable_batch.rb` | The Durable Batch: six billable "LLM calls" run under an ExecutionJournal. Mid-batch, the process dies for real - exit!,... | +| `error_taxonomy_drill.rb` | The Error Taxonomy Drill: three tasks fail three different ways - a rate limit (retryable, says the error itself), an au... | +| `exquisite_corpse.rb` | The Exquisite Corpse: three artists each draw one part of a creature without seeing the others' work; the assembler rece... | +| `flaky_api_drill.rb` | The Flaky API Drill: a task that times out twice before succeeding, run under a retry policy with exponential backoff an... | +| `form_errors.rb` | The 422 Generator: turn a ValidationError into the API error document your frontend actually wants - message, allowed va... | +| `freight_rules.rb` | The Freight Desk: a quoting capability whose tariff book is written as cross-field contract rules (new this round). Per-... | +| `gem_scout.rb` | Gem Scout: describe what you need, get a ranked shortlist of gems. Search and scoring are separate capabilities; the sea... | +| `graph_critic.rb` | The Graph Critic: reviews a plan's dependency structure BEFORE it runs, the way you'd review a class diagram. God tasks,... | +| `haiku_agent.rb` | The three-line agent. Run me with no API key at all: | +| `invariant_sentinel.rb` | The Invariant Sentinel: domain invariants checked after EVERY task, from a lifecycle hook. When a task leaves the world ... | +| `kanban_board.rb` | The Kanban Board: a plan rendered as the three columns everyone actually understands - To Do, Doing, Done - reprinted at... | +| `knee_finder.rb` | The Knee Finder: runs the same plan at increasing concurrency limits, measures wall time and total queue-wait via the ta... | +| `latency_lab.rb` | The Latency Lab: 20 simulated LLM calls (200ms of IO each) executed through the orchestrator at different concurrency li... | +| `live_dashboard.rb` | The Live Dashboard: lifecycle hooks publish events onto an Async::Queue; a consumer task IN THE SAME REACTOR renders the... | +| `namespace_cartographer.rb` | The Namespace Cartographer: maps a gem's constant tree and audits every file against the constant Zeitwerk expects it to... | +| `performance_detective.rb` | The Performance Detective: one task per Ruby file in lib/, fanned out through the orchestrator, each dissecting a file f... | +| `plan_diagram.rb` | The Plan Diagrammer: any orchestrator's graph, emitted as Mermaid - paste it into a README, GitHub renders it, and the d... | +| `plan_gantt.rb` | The Plan Gantt: lifecycle hooks timestamp every task, then the run is rendered as an ASCII timeline - where your wall cl... | +| `readme_verifier.rb` | The README Verifier: every ruby code fence in the README is a promise. This extracts them all, syntax-checks each with P... | +| `refactoring_dojo.rb` | The Refactoring Dojo: a student submits a method, three critic agents review it from three distinct perspectives, and th... | +| `renga_circle.rb` | A renga circle: three poet agents compose a linked-verse poem, each verse responding to the one before it. The dependenc... | +| `schema_advisor.rb` | The Schema Advisor: give it a schema and a query log, get back the advisories a careful DBA would write - each rule its ... | +| `setup_doctor.rb` | The Setup Doctor: every onboarding wiki page is a bug. This runs the checks a README asks a new hire to do by hand - rub... | +| `shared_rate_limit.rb` | The Shared Rate Limit: two plans run concurrently in one reactor, but the API key they share allows only 3 requests in f... | +| `stampede_sim.rb` | The Stampede Simulator: twenty workers hit a hiccuping upstream, all fail at once, all retry. With jitter OFF they come ... | +| `standup_digest.rb` | The Standup Digest: three collectors gather from the repo in parallel - recent commits, TODO debt, test suite shape - an... | +| `state_machine.rb` | The Contract State Machine: each transition is a capability whose guard is not an if-statement but an enum predicate on ... | +| `telephone_game.rb` | The telephone game: a rumor passes through five villagers, each of whom hears the previous version through the orchestra... | +| `three_shapes.rb` | Three Shapes: the same six units of work arranged three ways - a chain, a star, and staged joins - then measured and cri... | +| `ticket_screener.rb` | A HEY-style ticket screener: every inbound support ticket flows through screen -> categorize -> draft, all tickets in pa... | +| `typed_pipeline.rb` | A typed ETL pipeline: extract -> transform -> load, each stage a capability with a declared contract, composed into one ... | diff --git a/examples/examples_index.rb b/examples/examples_index.rb new file mode 100644 index 0000000..e14ccd0 --- /dev/null +++ b/examples/examples_index.rb @@ -0,0 +1,77 @@ +# frozen_string_literal: true + +# The Index Generator: reads every example's own header comment and +# writes examples/README.md - a catalog that regenerates from the +# files, so it can't rot. Complains about any example missing a run +# line, because an example you can't run is a rumor. +# +# bundle exec ruby examples/examples_index.rb +# +# Runs offline; rewrites examples/README.md in place. + +require_relative "../lib/agentic" + +DIR = __dir__ + +orchestrator = Agentic::PlanOrchestrator.new(concurrency_limit: 8) + +files = Dir[File.join(DIR, "*.rb")].sort - [File.join(DIR, File.basename(__FILE__))] + +surveys = files.map do |path| + task = Agentic::Task.new( + description: File.basename(path), + agent_spec: {"name" => "Librarian", "instructions" => "catalog the example"}, + payload: path + ) + orchestrator.add_task(task, agent: ->(t) { + lines = File.readlines(t.payload, encoding: "UTF-8") + .reject { |l| l.start_with?("# frozen_string_literal") } + header = lines.drop_while { |l| !l.start_with?("#") }.take_while { |l| l.start_with?("#") } + prose = header.map { |l| l.sub(/\A#\s?/, "").rstrip } + run_line = prose.find { |l| l.strip.start_with?("bundle exec ruby") } + + { + file: File.basename(t.payload), + summary: prose.take_while { |l| !l.empty? }.join(" ").squeeze(" "), + runnable: !run_line.nil? + } + }) + task +end + +index = Agentic::Task.new( + description: "write the index", + agent_spec: {"name" => "Editor", "instructions" => "assemble the catalog"} +) +orchestrator.add_task(index, surveys, agent: ->(t) { + entries = surveys.map { |s| t.output_of(s) }.sort_by { |e| e[:file] } + missing_run = entries.reject { |e| e[:runnable] } + + lines = ["# Examples", ""] + lines << "Every example runs offline: `bundle exec ruby examples/`." + lines << "This index is generated by `examples_index.rb`; edit the examples," + lines << "not this file." + lines << "" + lines << "| Example | What it shows |" + lines << "|---------|---------------|" + entries.each do |entry| + summary = entry[:summary][0, 120] + summary += "..." if entry[:summary].length > 120 + lines << "| `#{entry[:file]}` | #{summary} |" + end + {markdown: lines.join("\n") + "\n", count: entries.size, missing_run: missing_run.map { |e| e[:file] }} +}) + +result = orchestrator.execute_plan +catalog = result.results[index.id].output + +File.write(File.join(DIR, "README.md"), catalog[:markdown]) + +puts "EXAMPLES INDEX: #{catalog[:count]} examples cataloged into examples/README.md" +if catalog[:missing_run].any? + puts " examples missing a run line (fix their headers):" + catalog[:missing_run].each { |f| puts " - #{f}" } + exit 1 +else + puts " every example documents how to run it. catalog is honest." +end From 5f805656e7ee94a81a472af5c582992837f92345 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 6 Jul 2026 21:35:42 +0000 Subject: [PATCH 12/36] docs: round-5 index and findings Adds the round-5 table to the perspectives README: the ecosystem turn - four tools built on the one-round-old graph accessor, every round-4 feature characterized under load, and the next asks (graph order and labeled edges, structured rule identifiers, full jitter, windowed rate limits). Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01FeZLdZ3M4q4cho4DZPfSkF --- docs/perspectives/README.md | 41 +++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+) diff --git a/docs/perspectives/README.md b/docs/perspectives/README.md index 486f826..840653d 100644 --- a/docs/perspectives/README.md +++ b/docs/perspectives/README.md @@ -85,6 +85,47 @@ consulting `failure.retryable?`, and contract value predicates — | 9 | Sandi Metz | Graph critic — design review for dependency graphs, pre-execution | `examples/graph_critic.rb` | [round-4/09-sandimetz.md](round-4/09-sandimetz.md) | | 10 | Andrew Kane | README verifier — every snippet parsed, every constant resolved (found a 4-round-old broken snippet) | `examples/readme_verifier.rb` | [round-4/10-ankane.md](round-4/10-ankane.md) | +## Round 5 — the ecosystem turn + +The round-4 asks shipped as a release (`PlanOrchestrator#graph`, +`ValidationError#expectations`, cross-field contract `rules:`, +`Agentic::RateLimit` + `LlmClient limiter:`, jitter-on-by-default), the +three examples that requested them were modernized onto them, and ten +more experiments followed: + +| # | Persona | Built on the round-5 release | Run it | Field notes | +|---|---------|------------------------------|--------|-------------| +| 1 | Matz | Dungeon crawl — the map drawn from the plan itself | `examples/dungeon_crawl.rb` | [round-5/01-matz.md](round-5/01-matz.md) | +| 2 | DHH | Live kanban — the WIP limit is the concurrency limit | `examples/kanban_board.rb` | [round-5/02-dhh.md](round-5/02-dhh.md) | +| 3 | Aaron Patterson | Critical path — which task the wall clock is actually about | `examples/critical_path.rb` | [round-5/03-tenderlove.md](round-5/03-tenderlove.md) | +| 4 | Xavier Noria | Mermaid diagrammer — docs generated from the graph, labeled by `needs:` | `examples/plan_diagram.rb` | [round-5/04-fxn.md](round-5/04-fxn.md) | +| 5 | Samuel Williams | Burst absorber — `RateLimit` characterized under hostile waves | `examples/burst_absorber.rb` | [round-5/05-ioquatix.md](round-5/05-ioquatix.md) | +| 6 | Jeremy Evans | Freight desk — a tariff book as cross-field rules, all violations at once | `examples/freight_rules.rb` | [round-5/06-jeremyevans.md](round-5/06-jeremyevans.md) | +| 7 | Piotr Solnica | 422 generator — one contract-agnostic error renderer from `expectations` | `examples/form_errors.rb` | [round-5/07-solnic.md](round-5/07-solnic.md) | +| 8 | Mike Perham | Stampede simulator — the jitter default, argued by histogram | `examples/stampede_sim.rb` | [round-5/08-mperham.md](round-5/08-mperham.md) | +| 9 | Sandi Metz | Three shapes — chain vs star vs staged, chosen by evidence | `examples/three_shapes.rb` | [round-5/09-sandimetz.md](round-5/09-sandimetz.md) | +| 10 | Andrew Kane | Examples index — self-maintaining signage for a 40-example gallery | `examples/examples_index.rb` | [round-5/10-ankane.md](round-5/10-ankane.md) | + +### What round 5 surfaced + +1. **The graph accessor compounded immediately**: one round old, it fed + a game map, a critical-path analyzer, a Mermaid generator, and a + design curriculum. Expose the right projection and an ecosystem + assembles itself. +2. **Named dependencies turned out to be documentation**: `needs:` + labels became labeled diagram edges — ergonomics maturing into + architecture records. +3. **Every round-4 feature was characterized under load the round it + shipped** — the burst absorber (RateLimit), the stampede histogram + (jitter), the freight desk (rules), the 422 generator + (expectations). +4. **Next asks**: `graph[:order]` (topological sort — requested + independently by three personas) plus `graph[:edges]` with labels, + structured rule identifiers (`{rule: :symbol, fields: [...]}`) so + policy violations can point at widgets, a `backoff_jitter: :full` + tier, and time-windowed rate limits alongside the concurrency + ceiling. + ### What round 4 surfaced 1. **Two more real defects found by examples**: canceled plans reported From 633278ac0c51103a86390dcd04c93c5480ae153e Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 7 Jul 2026 01:22:49 +0000 Subject: [PATCH 13/36] feat: deliver the round-5 asks The prioritized asks from the round-5 field notes, implemented: - graph[:order]: topological sort via Kahn's algorithm (cycle members appended rather than dropped) - requested independently by three personas building graph tools - graph[:edges]: positional and named dependencies pre-merged into frozen {from:, to:, label:} records, labels from needs: names - Structured cross-field rules: rules may declare {message:, fields:, check:} keyed by identifier; prose-form rules keep working. ValidationError#rule_violations reports each broken rule with id, message, and the fields it reads, so UIs can highlight offending inputs - backoff_jitter: :full draws delays uniformly from [0, delay] for harder herd-flattening than equal jitter - Windowed rate limits: RateLimit.new(30, per: 60) admits at most ceiling acquisitions per rolling window (quota physics), alongside the existing concurrency mode (connection physics) Modernized onto the new APIs: plan_diagram (edges + order), dungeon_crawl (map in delve order), freight_rules (structured rules), form_errors (policy violations highlight fields). Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01FeZLdZ3M4q4cho4DZPfSkF --- README.md | 18 +++- examples/dungeon_crawl.rb | 5 +- examples/form_errors.rb | 11 +- examples/freight_rules.rb | 37 +++++-- examples/plan_diagram.rb | 23 ++-- lib/agentic/capability_validator.rb | 23 +++- lib/agentic/errors.rb | 10 +- lib/agentic/plan_orchestrator.rb | 45 +++++++- lib/agentic/rate_limit.rb | 55 ++++++++-- spec/agentic/round6_features_spec.rb | 155 +++++++++++++++++++++++++++ 10 files changed, 328 insertions(+), 54 deletions(-) create mode 100644 spec/agentic/round6_features_spec.rb diff --git a/README.md b/README.md index f0fdf98..2a73b25 100644 --- a/README.md +++ b/README.md @@ -244,11 +244,19 @@ The `task_slot_acquired` lifecycle hook fires when a task obtains a concurrency slot (with `waited:` time), separating queue wait from run time; retry policies consult an error's own `retryable?` verdict before falling back to the `retryable_errors` list, and backoff jitter is on by -default so fleets don't retry in lockstep. `PlanOrchestrator#graph` -returns a frozen snapshot of the plan's topology for tools that render or -review it, and `Agentic::RateLimit` is a credential-scoped concurrency -ceiling shareable across plans and clients -(`Agentic::LlmClient.new(config, limiter: rate_limit)`). +default so fleets don't retry in lockstep (`backoff_jitter: :full` draws +uniformly from `[0, delay]` for the hardest herd-flattening). + +`PlanOrchestrator#graph` returns a frozen snapshot of the plan's topology +— including `graph[:order]` (topological sort) and `graph[:edges]` +(labeled by `needs:` names) — for tools that render, review, or analyze +plans. `Agentic::RateLimit` is a credential-scoped ceiling shareable +across plans and clients (`Agentic::LlmClient.new(config, limiter: rate_limit)`): +concurrent by default, or a rolling-window quota with +`RateLimit.new(30, per: 60)`. Cross-field rules may be structured — +`rules: {air_weight_limit: {message: "...", fields: [:mode, :weight], check: ->(i) {...}}}` — +and `ValidationError#rule_violations` reports each broken rule with its +identifier, message, and the fields it reads. ### The concurrency contract diff --git a/examples/dungeon_crawl.rb b/examples/dungeon_crawl.rb index 1caec78..0b04456 100644 --- a/examples/dungeon_crawl.rb +++ b/examples/dungeon_crawl.rb @@ -51,9 +51,10 @@ def room(name) graph = orchestrator.graph names = graph[:tasks].transform_values(&:description) -puts "THE MAP (drawn from orchestrator.graph, before anyone delves)" +puts "THE MAP (drawn from orchestrator.graph, in delving order)" puts -graph[:dependencies].each do |room_id, door_ids| +graph[:order].each do |room_id| + door_ids = graph[:dependencies][room_id] if door_ids.empty? puts " [#{names[room_id]}] <- you are here" else diff --git a/examples/form_errors.rb b/examples/form_errors.rb index 0d601c1..472edc7 100644 --- a/examples/form_errors.rb +++ b/examples/form_errors.rb @@ -24,7 +24,11 @@ coupon: {type: "string"} }, rules: { - "starter plan is limited to 5 seats" => ->(i) { i[:plan] != "starter" || i[:seats] <= 5 } + starter_seat_limit: { + message: "starter plan is limited to 5 seats", + fields: [:plan, :seats], + check: ->(i) { i[:plan] != "starter" || i[:seats] <= 5 } + } } ) Agentic.register_capability(spec, Agentic::CapabilityProvider.new( @@ -50,7 +54,10 @@ def error_document(error) status: 422, capability: error.capability, errors: field_errors, - policy_violations: error.violations.fetch(:base, []) + # Structured rule violations point at the widgets they involve + policy_violations: error.rule_violations.map { |v| + {rule: v[:rule], message: v[:message], highlight_fields: v[:fields]} + } } end diff --git a/examples/freight_rules.rb b/examples/freight_rules.rb index 605f96a..4fa115a 100644 --- a/examples/freight_rules.rb +++ b/examples/freight_rules.rb @@ -24,14 +24,26 @@ destination: {type: "string", required: true, non_empty: true} }, rules: { - "air freight is limited to 500kg" => - ->(i) { i[:mode] != "air" || i[:weight_kg] <= 500 }, - "hazardous cargo may not fly" => - ->(i) { !(i[:mode] == "air" && i[:hazardous]) }, - "insured value over 100k requires sea mode" => - ->(i) { i[:insured_value] <= 100_000 || i[:mode] == "sea" }, - "road freight only reaches domestic destinations" => - ->(i) { i[:mode] != "road" || i[:destination].start_with?("domestic:") } + air_weight_limit: { + message: "air freight is limited to 500kg", + fields: [:mode, :weight_kg], + check: ->(i) { i[:mode] != "air" || i[:weight_kg] <= 500 } + }, + no_hazardous_air: { + message: "hazardous cargo may not fly", + fields: [:mode, :hazardous], + check: ->(i) { !(i[:mode] == "air" && i[:hazardous]) } + }, + high_value_by_sea: { + message: "insured value over 100k requires sea mode", + fields: [:mode, :insured_value], + check: ->(i) { i[:insured_value] <= 100_000 || i[:mode] == "sea" } + }, + road_is_domestic: { + message: "road freight only reaches domestic destinations", + fields: [:mode, :destination], + check: ->(i) { i[:mode] != "road" || i[:destination].start_with?("domestic:") } + } } ) @@ -57,9 +69,12 @@ quote = desk.execute(manifest) puts format(" #%d QUOTED $%.2f (%s, %dkg)", index + 1, quote[:quote], manifest[:mode], manifest[:weight_kg]) rescue Agentic::Errors::ValidationError => e - if e.violations.key?(:base) - puts " ##{index + 1} REFUSED - #{e.violations[:base].size} rule(s) broken:" - e.violations[:base].each { |rule| puts " - #{rule}" } + if e.rule_violations.any? + puts " ##{index + 1} REFUSED - #{e.rule_violations.size} rule(s) broken:" + e.rule_violations.each do |violation| + puts " - [#{violation[:rule]}] #{violation[:message]} " \ + "(fields: #{violation[:fields].join(", ")})" + end else puts " ##{index + 1} MALFORMED - #{e.violations.keys.join(", ")} invalid " \ "(never reached the tariff book)" diff --git a/examples/plan_diagram.rb b/examples/plan_diagram.rb index 7ea87d5..828b9c1 100644 --- a/examples/plan_diagram.rb +++ b/examples/plan_diagram.rb @@ -31,26 +31,17 @@ def step(name) orchestrator.add_task(publish, [draft]) # --- the diagrammer: graph in, mermaid out ----------------------------------- +# graph[:edges] arrives pre-merged with labels and graph[:order] gives +# stable, topological node numbering - both were this example's asks. def to_mermaid(graph) names = graph[:tasks].transform_values(&:description) - ids = names.keys.each_with_index.to_h { |task_id, i| [task_id, "T#{i}"] } - - named_edges = graph[:needs].flat_map { |task_id, needs| - needs.map { |label, dep_id| [dep_id, task_id, label] } - } - labeled = named_edges.to_h { |from, to, _| [[from, to], true] } + ids = graph[:order].each_with_index.to_h { |task_id, i| [task_id, "T#{i}"] } lines = ["graph TD"] - names.each { |task_id, name| lines << " #{ids[task_id]}[\"#{name}\"]" } - graph[:dependencies].each do |task_id, deps| - deps.each do |dep_id| - next if labeled[[dep_id, task_id]] # named edges are drawn with labels below - - lines << " #{ids[dep_id]} --> #{ids[task_id]}" - end - end - named_edges.each do |from, to, label| - lines << " #{ids[from]} -- #{label} --> #{ids[to]}" + graph[:order].each { |task_id| lines << " #{ids[task_id]}[\"#{names[task_id]}\"]" } + graph[:edges].each do |edge| + arrow = edge[:label] ? "-- #{edge[:label]} -->" : "-->" + lines << " #{ids[edge[:from]]} #{arrow} #{ids[edge[:to]]}" end lines.join("\n") end diff --git a/lib/agentic/capability_validator.rb b/lib/agentic/capability_validator.rb index 2429554..967a176 100644 --- a/lib/agentic/capability_validator.rb +++ b/lib/agentic/capability_validator.rb @@ -62,18 +62,35 @@ def validate!(kind, declared, values) end # Cross-field rules run after per-key validation, over the whole - # inputs hash; every broken rule is reported at once under :base + # inputs hash; every broken rule is reported at once. Rules come in + # two forms: + # + # "prose description" => ->(inputs) { ... } # simple + # :rule_id => {message: "...", fields: [:a, :b], check: ->} # structured + # + # Structured rules declare which fields they read, so violations can + # point UIs at the offending inputs. def validate_rules!(inputs) rules = @specification.respond_to?(:rules) ? @specification.rules : nil return if rules.nil? || rules.empty? - broken = rules.reject { |_description, check| check.call(inputs) }.keys + broken = rules.filter_map do |key, definition| + check, message, fields = + if definition.respond_to?(:call) + [definition, key.to_s, []] + else + [definition.fetch(:check), definition[:message] || key.to_s, definition[:fields] || []] + end + + {rule: key, message: message, fields: fields} unless check.call(inputs) + end return if broken.empty? raise Errors::ValidationError.new( capability: @specification.name, kind: :inputs, - violations: {base: broken} + violations: {base: broken.map { |b| b[:message] }}, + rule_violations: broken ) end diff --git a/lib/agentic/errors.rb b/lib/agentic/errors.rb index 643f0da..c02010e 100644 --- a/lib/agentic/errors.rb +++ b/lib/agentic/errors.rb @@ -32,15 +32,23 @@ class ValidationError < StandardError # @return [Hash{Symbol=>Hash}] Declarations keyed by violated attribute attr_reader :expectations + # Structured cross-field rule violations: each carries the rule's + # identifier, its human message, and the fields it reads - so UIs + # can highlight the offending inputs, not just print prose + # @return [Array] [{rule:, message:, fields:}, ...] + attr_reader :rule_violations + # @param capability [String] The capability name # @param kind [Symbol] :inputs or :outputs # @param violations [Hash{Symbol=>Array}] Messages keyed by attribute # @param expectations [Hash{Symbol=>Hash}] Declarations for violated keys - def initialize(capability:, kind:, violations:, expectations: {}) + # @param rule_violations [Array] Structured broken-rule records + def initialize(capability:, kind:, violations:, expectations: {}, rule_violations: []) @capability = capability @kind = kind @violations = violations @expectations = expectations + @rule_violations = rule_violations details = violations.map { |key, messages| "#{key} #{Array(messages).join(", ")}" }.join("; ") super("Invalid #{kind} for capability '#{capability}': #{details}") diff --git a/lib/agentic/plan_orchestrator.rb b/lib/agentic/plan_orchestrator.rb index 3c33da0..924ee6d 100644 --- a/lib/agentic/plan_orchestrator.rb +++ b/lib/agentic/plan_orchestrator.rb @@ -97,12 +97,22 @@ def add_task(task, dependencies = [], agent: nil, needs: nil) # A read-only snapshot of the plan's topology, for tools that render, # review, or analyze the graph without executing it # @return [Hash] :tasks (id => Task), :dependencies (id => [ids]), - # :needs (id => {name => id}) + # :needs (id => {name => id}), :order (task ids, topologically + # sorted), :edges ([{from:, to:, label:}] with labels from needs:) def graph + labels = @task_needs.each_with_object({}) do |(task_id, named), acc| + named.each { |name, dep_id| acc[[dep_id, task_id]] = name } + end + edges = @dependencies.flat_map { |task_id, deps| + deps.map { |dep_id| {from: dep_id, to: task_id, label: labels[[dep_id, task_id]]}.freeze } + }.freeze + { tasks: @tasks.dup.freeze, dependencies: @dependencies.transform_values { |deps| deps.dup.freeze }.freeze, - needs: @task_needs.transform_values { |named| named.dup.freeze }.freeze + needs: @task_needs.transform_values { |named| named.dup.freeze }.freeze, + order: topological_order.freeze, + edges: edges }.freeze end @@ -257,8 +267,13 @@ def apply_retry_backoff(task:) 0 end - # Apply jitter (on by default) so fleets don't retry in lockstep - if @retry_policy[:backoff_jitter] + # Apply jitter (on by default) so fleets don't retry in lockstep. + # true = equal jitter (+/-25%); :full = full jitter (uniform over + # [0, delay]), which flattens synchronized herds much harder + case @retry_policy[:backoff_jitter] + when :full + delay = rand(0.0..delay) + when true jitter_factor = 0.25 # Default 25% jitter jitter = rand(-delay * jitter_factor..delay * jitter_factor) delay = [delay + jitter, 0].max @@ -307,6 +322,28 @@ def overall_status private + # Kahn's algorithm over the declared dependencies. Tasks caught in a + # dependency cycle are appended (in insertion order) after the sorted + # portion rather than silently dropped. + # @return [Array] Task ids, dependencies before dependents + def topological_order + remaining_deps = @dependencies.transform_values(&:dup) + order = [] + ready = remaining_deps.select { |_, deps| deps.empty? }.keys + + until ready.empty? + task_id = ready.shift + order << task_id + remaining_deps.each do |candidate, deps| + next unless deps.delete(task_id) + + ready << candidate if deps.empty? && !order.include?(candidate) + end + end + + order + (@dependencies.keys - order) + end + # Schedules a task for execution using the semaphore to limit concurrency # @param task_id [String] ID of the task to schedule # @param agent_provider [Object] Provides agents for task execution diff --git a/lib/agentic/rate_limit.rb b/lib/agentic/rate_limit.rb index cfa6587..105766c 100644 --- a/lib/agentic/rate_limit.rb +++ b/lib/agentic/rate_limit.rb @@ -22,10 +22,20 @@ class RateLimit # @return [Integer] The most concurrent acquisitions observed attr_reader :high_water - # @param ceiling [Integer] Maximum concurrent acquisitions - def initialize(ceiling) + # @return [Numeric, nil] The rolling window in seconds, when windowed + attr_reader :per + + # @param ceiling [Integer] Maximum acquisitions - concurrent when + # per: is nil, per rolling window when per: is given + # @param per [Numeric, nil] Window length in seconds. A concurrency + # ceiling models connection limits (3 requests in flight); a + # windowed ceiling models quota (30 requests per minute). They are + # different physics; pick the one your provider enforces. + def initialize(ceiling, per: nil) @ceiling = ceiling - @semaphore = Async::Semaphore.new(ceiling) + @per = per + @semaphore = Async::Semaphore.new(ceiling) unless per + @stamps = [] @in_flight = 0 @high_water = 0 end @@ -34,18 +44,43 @@ def initialize(ceiling) # @yield The rate-limited work # @return [Object] The block's return value def acquire + return windowed_acquire { yield } if @per + @semaphore.acquire do - @in_flight += 1 - @high_water = [@high_water, @in_flight].max - begin - yield - ensure - @in_flight -= 1 - end + track { yield } end end # @return [Integer] Acquisitions currently inside the ceiling attr_reader :in_flight + + private + + def track + @in_flight += 1 + @high_water = [@high_water, @in_flight].max + yield + ensure + @in_flight -= 1 + end + + # Rolling-window admission: wait until fewer than ceiling + # acquisitions have started within the last `per` seconds + def windowed_acquire + loop do + now = clock + @stamps.reject! { |stamp| stamp <= now - @per } + break if @stamps.size < @ceiling + + sleep(@stamps.first + @per - now) + end + + @stamps << clock + track { yield } + end + + def clock + Process.clock_gettime(Process::CLOCK_MONOTONIC) + end end end diff --git a/spec/agentic/round6_features_spec.rb b/spec/agentic/round6_features_spec.rb new file mode 100644 index 0000000..240538b --- /dev/null +++ b/spec/agentic/round6_features_spec.rb @@ -0,0 +1,155 @@ +# frozen_string_literal: true + +require "spec_helper" + +RSpec.describe "round 6 framework features" do + def task_named(description) + Agentic::Task.new( + description: description, + agent_spec: {"name" => "worker", "instructions" => "work"} + ) + end + + describe "graph[:order] and graph[:edges]" do + it "returns task ids topologically sorted regardless of insertion order" do + orchestrator = Agentic::PlanOrchestrator.new + first = task_named("first") + middle = task_named("middle") + last = task_named("last") + + # Deliberately inserted backwards + orchestrator.add_task(last, [middle], agent: ->(_t) { :ok }) + orchestrator.add_task(middle, [first], agent: ->(_t) { :ok }) + orchestrator.add_task(first, agent: ->(_t) { :ok }) + + expect(orchestrator.graph[:order]).to eq([first.id, middle.id, last.id]) + end + + it "merges positional and named dependencies into labeled edges" do + orchestrator = Agentic::PlanOrchestrator.new + gate = task_named("gate") + source = task_named("source") + sink = task_named("sink") + + orchestrator.add_task(gate, agent: ->(_t) { :ok }) + orchestrator.add_task(source, agent: ->(_t) { :ok }) + orchestrator.add_task(sink, [gate], needs: {input: source}, agent: ->(_t) { :ok }) + + edges = orchestrator.graph[:edges] + + expect(edges).to contain_exactly( + {from: gate.id, to: sink.id, label: nil}, + {from: source.id, to: sink.id, label: :input} + ) + expect(edges).to be_frozen + end + + it "appends cycle members after the sorted portion instead of dropping them" do + orchestrator = Agentic::PlanOrchestrator.new + a = task_named("a") + b = task_named("b") + free = task_named("free") + + orchestrator.add_task(a, [b], agent: ->(_t) { :ok }) + orchestrator.add_task(b, [a], agent: ->(_t) { :ok }) + orchestrator.add_task(free, agent: ->(_t) { :ok }) + + order = orchestrator.graph[:order] + + expect(order.first).to eq(free.id) + expect(order).to contain_exactly(free.id, a.id, b.id) + end + end + + describe "structured rules" do + let(:specification) do + Agentic::CapabilitySpecification.new( + name: "quote", description: "quotes", version: "1.0.0", + inputs: { + mode: {type: "string", required: true}, + weight: {type: "number", required: true} + }, + rules: { + :air_weight_limit => { + message: "air freight is limited to 500kg", + fields: [:mode, :weight], + check: ->(i) { i[:mode] != "air" || i[:weight] <= 500 } + }, + "prose-style rules still work" => ->(i) { i[:weight] >= 1 } + } + ) + end + + it "reports structured violations with rule id, message, and fields" do + validator = Agentic::CapabilityValidator.new(specification) + + expect { + validator.validate_inputs!(mode: "air", weight: 900) + }.to raise_error(Agentic::Errors::ValidationError) { |error| + expect(error.rule_violations).to eq([ + {rule: :air_weight_limit, message: "air freight is limited to 500kg", fields: [:mode, :weight]} + ]) + expect(error.violations[:base]).to eq(["air freight is limited to 500kg"]) + } + end + + it "keeps prose rules working alongside structured ones" do + validator = Agentic::CapabilityValidator.new(specification) + + expect { + validator.validate_inputs!(mode: "sea", weight: 0) + }.to raise_error(Agentic::Errors::ValidationError) { |error| + expect(error.rule_violations).to eq([ + {rule: "prose-style rules still work", message: "prose-style rules still work", fields: []} + ]) + } + end + end + + describe "full jitter" do + it "draws delays uniformly from [0, delay]" do + orchestrator = Agentic::PlanOrchestrator.new( + retry_policy: {backoff_strategy: :constant, backoff_constant: 1.0, backoff_jitter: :full} + ) + slept = [] + allow(orchestrator).to receive(:sleep) { |delay| slept << delay } + + task = task_named("flaky") + task.retry_count = 1 + 50.times { orchestrator.apply_retry_backoff(task: task) } + + expect(slept).to all(be_between(0.0, 1.0)) + expect(slept.count { |d| d < 0.5 }).to be > 5 # full jitter reaches low values + end + end + + describe "windowed RateLimit" do + it "admits at most ceiling acquisitions per rolling window" do + limit = Agentic::RateLimit.new(3, per: 0.1) + stamps = [] + + Sync do + 6.times.map { + Async do + limit.acquire { stamps << Process.clock_gettime(Process::CLOCK_MONOTONIC) } + end + }.each(&:wait) + end + + windows = stamps.sort + # The 4th acquisition must start at least one window after the 1st + expect(windows[3] - windows[0]).to be >= 0.09 + expect(stamps.size).to eq(6) + end + + it "keeps concurrency mode unchanged when per: is omitted" do + limit = Agentic::RateLimit.new(2) + + Sync do + 6.times.map { Async { limit.acquire { sleep(0.01) } } }.each(&:wait) + end + + expect(limit.high_water).to eq(2) + end + end +end From 9faaf099c1da04d44ba127d2fbb70c41b57f1d26 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 7 Jul 2026 01:24:22 +0000 Subject: [PATCH 14/36] docs: plan tour example (Matz round 6) Any plan narrated as prose from graph[:order] and graph[:edges] - needs: labels surface as purpose, and reading the plan aloud catches sequence bugs before execution. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01FeZLdZ3M4q4cho4DZPfSkF --- docs/perspectives/round-6/01-matz.md | 55 ++++++++++++++++++++++ examples/plan_tour.rb | 68 ++++++++++++++++++++++++++++ 2 files changed, 123 insertions(+) create mode 100644 docs/perspectives/round-6/01-matz.md create mode 100644 examples/plan_tour.rb diff --git a/docs/perspectives/round-6/01-matz.md b/docs/perspectives/round-6/01-matz.md new file mode 100644 index 0000000..8bad905 --- /dev/null +++ b/docs/perspectives/round-6/01-matz.md @@ -0,0 +1,55 @@ +# Round 6 field notes — Matz narrates breakfast + +*Built: `examples/plan_tour.rb` — any plan, narrated as prose from +`graph[:order]` and `graph[:edges]`, before anything runs.* + +## What I built and why + +I asked for `graph[:order]` so map-drawers could be correct on purpose; +it arrived, and the first thing I built with it is the oldest debugging +technique there is — **reading your program aloud**: + +``` +First, boil the kettle. +Meanwhile, slice the bread. +After "boil the kettle": soft-boil the eggs. +After "soft-boil the eggs" (your protein), "toast the bread" (your + crunch) and "steep the tea" (your comfort): plate everything. +``` + +Every word is generated: "First" is the order's head, "Meanwhile" is a +root task appearing later in the order, "After X:" is an edge, and +"(your protein)" is a `needs:` label surfacing as *purpose*. If the +prose sounds wrong — if breakfast steeps the tea before boiling the +kettle — the plan is wrong, and your ears caught it while the stove +was still cold. Eyes skim structure; ears parse sequence. Rubber-duck +debugging works because speech serializes thought, and now the +framework serializes the plan for you. + +## Confession, in the round's tradition + +My first narrator said "Once you boil the kettle is done" — I tried to +inflect English with a regex and produced grammar only a parser could +love. Aaron's law extends past Ruby: *don't parse natural language +with regexes either, not even outbound.* Quoting the task names +verbatim ("After \"boil the kettle\":") is humbler and better — the +narration frames; the plan speaks in its own words. (This is also, +quietly, the correct division of labor for the LLM upgrade: let a +model smooth the prose, but generate the *skeleton* from the graph so +the sequence can never be hallucinated.) + +## Small notes + +- `order` + `edges` is the whole API surface this needed. Two rounds + ago this program would have been mostly graph traversal; now it is + mostly sentences. The framework's share of my programs keeps + shrinking, which is the trend line I care about most. +- The `needs:` labels reading as "(your protein)" delighted me — the + consumer named the dependency for code clarity in round 3, and by + round 6 the same name explains the *cuisine*. Names are the gift + that keeps arriving. + +## Verdict + +The plan can draw itself (round 5) and now speak for itself (round 6). +Next it should probably listen — but that's another round. diff --git a/examples/plan_tour.rb b/examples/plan_tour.rb new file mode 100644 index 0000000..aa2f022 --- /dev/null +++ b/examples/plan_tour.rb @@ -0,0 +1,68 @@ +# frozen_string_literal: true + +# The Plan Tour: hand any orchestrator to the guide and it narrates the +# plan as prose - first this, then that, meanwhile the other - read +# straight from graph[:order] and graph[:edges]. If the prose sounds +# wrong, your plan IS wrong, and you found out before running it. +# +# bundle exec ruby examples/plan_tour.rb +# +# Runs offline; narration only, no execution. + +require_relative "../lib/agentic" + +# A breakfast, planned properly +orchestrator = Agentic::PlanOrchestrator.new + +def step(name) + Agentic::Task.new(description: name, agent_spec: {"name" => name, "instructions" => "cook"}) +end + +kettle = step("boil the kettle") +eggs = step("soft-boil the eggs") +bread = step("slice the bread") +toast = step("toast the bread") +tea = step("steep the tea") +plate = step("plate everything") + +orchestrator.add_task(kettle) +orchestrator.add_task(bread) +orchestrator.add_task(eggs, [kettle]) +orchestrator.add_task(tea, [kettle]) +orchestrator.add_task(toast, [bread]) +orchestrator.add_task(plate, needs: {protein: eggs, crunch: toast, comfort: tea}) + +# --- the guide: graph in, prose out ------------------------------------------ +def narrate(graph) + names = graph[:tasks].transform_values(&:description) + incoming = graph[:edges].group_by { |e| e[:to] } + narrated = [] + sentences = [] + + graph[:order].each do |task_id| + edges_in = incoming[task_id] || [] + sentence = + if edges_in.empty? + opener = narrated.empty? ? "First" : "Meanwhile" + "#{opener}, #{names[task_id]}." + else + reasons = edges_in.map { |e| + e[:label] ? "\"#{names[e[:from]]}\" (your #{e[:label]})" : "\"#{names[e[:from]]}\"" + } + joined = (reasons.size > 1) ? reasons[0..-2].join(", ") + " and #{reasons.last}" : reasons.first + "After #{joined}: #{names[task_id]}." + end + sentences << sentence + narrated << task_id + end + + sentences +end + +puts "THE PLAN, AS PROSE (nobody has cooked anything yet)" +puts +narrate(orchestrator.graph).each { |sentence| puts " #{sentence}" } +puts +puts "the narration is generated from graph[:order] and graph[:edges] -" +puts "the same topology the scheduler will execute. read your plan aloud" +puts "before you run it; ears catch what eyes skim." From 1ef0a1f3f2cd818732d8eca83193827b9fcebce8 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 7 Jul 2026 01:25:43 +0000 Subject: [PATCH 15/36] docs: deploy train example (DHH round 6) A gated pipeline run healthy and with a red canary: one after_task_failure hook is the brake, canceled cars are reported as CANCELED, and failure correctly outranks cancellation in the status. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01FeZLdZ3M4q4cho4DZPfSkF --- docs/perspectives/round-6/02-dhh.md | 57 ++++++++++++++++++++ examples/deploy_train.rb | 82 +++++++++++++++++++++++++++++ 2 files changed, 139 insertions(+) create mode 100644 docs/perspectives/round-6/02-dhh.md create mode 100644 examples/deploy_train.rb diff --git a/docs/perspectives/round-6/02-dhh.md b/docs/perspectives/round-6/02-dhh.md new file mode 100644 index 0000000..bb587ba --- /dev/null +++ b/docs/perspectives/round-6/02-dhh.md @@ -0,0 +1,57 @@ +# Round 6 field notes — DHH runs the deploy train + +*Built: `examples/deploy_train.rb` — lint → test → build → canary → +ship → announce, run healthy and then with a failing canary. The +second run is the product.* + +## What I built and why + +Every deploy pipeline demo shows the green path, which is like +reviewing a seatbelt by admiring the buckle. The train's whole reason +to exist is Friday: + +``` +canary RED - error rate 4.2% exceeds 1% threshold +ship CANCELED (never left the yard) +announce CANCELED (never left the yard) +train status: partial_failure +``` + +One `after_task_failure` hook calling `cancel_plan` is the entire +brake. The cars behind the red gate report **CANCELED** — a word, in +the results, queryable — not the silent absence that makes 2am +debugging a archaeology dig ("did announce run? grep the logs..."). +Jeremy's round-4 status fix is what makes this honest: before it, a +stopped train could report `:completed` and you'd announce a release +you never shipped. I have seen that exact incident in the wild. +Twice. + +## What I learned writing the copy + +My first closing line claimed the train status was `:canceled` — it's +actually `:partial_failure`, because failure outranks cancellation in +the status precedence. And sitting with it, that's *right*: the +headline should be WHY the train stopped (a gate went red); the +manifest shows what never shipped (the canceled cars). Two different +questions, answered at two different levels. I fixed my prose, not the +framework — a rarity worth recording, and the sign the status enum has +finally earned trust: when the tool and I disagreed, the tool was +right. + +## Omakase notes + +- `max_retries: 0` on the train. Deploy gates should not be retried + into submission; a flaky canary is a red canary. Retry budgets are + for *ingestion*, not *judgment*. +- `concurrency_limit: 1` because a deploy train is a train. There are + workloads where the queue IS the feature; the framework respecting + that without ceremony (one integer) is the omakase experience. +- This is `bin/deploy` for anyone whose deploy is Ruby-orchestrable. + Swap the sleeps for `system` calls and the canary check for a real + metrics query; the shape ships as-is. + +## Verdict + +The unhappy path is the product, and it now reads like a train +manifest instead of a shrug. Six stations, one hook, zero incident +reports that end with "we didn't realize it kept going." diff --git a/examples/deploy_train.rb b/examples/deploy_train.rb new file mode 100644 index 0000000..d238cb7 --- /dev/null +++ b/examples/deploy_train.rb @@ -0,0 +1,82 @@ +# frozen_string_literal: true + +# The Deploy Train: lint -> test -> build -> canary -> ship, where a +# red gate stops the train and everything behind it reports CANCELED, +# not skipped-and-shrugged. Run it twice: healthy train, then a canary +# failure. The second run is why deploy pipelines exist. +# +# bundle exec ruby examples/deploy_train.rb +# +# Runs offline; the canary's health is scripted. + +require_relative "../lib/agentic" + +Agentic.logger.level = :fatal # the failure is the demo, not news + +STATIONS = %w[lint test build canary ship announce].freeze + +def run_train(canary_healthy:) + orchestrator = nil + hooks = { + after_task_failure: ->(task_id:, task:, failure:, duration:) { + # A red gate stops the whole train - no half-shipped releases + orchestrator.cancel_plan + } + } + orchestrator = Agentic::PlanOrchestrator.new( + concurrency_limit: 1, + lifecycle_hooks: hooks, + retry_policy: {max_retries: 0, retryable_errors: []} + ) + + previous = nil + tasks = STATIONS.to_h do |station| + task = Agentic::Task.new( + description: station, + agent_spec: {"name" => station, "instructions" => "run the gate"} + ) + orchestrator.add_task(task, previous ? [previous] : [], agent: ->(t) { + sleep(0.01) + if t.description == "canary" && !canary_healthy + raise "error rate 4.2% exceeds 1% threshold" + end + + :green + }) + previous = task + [station, task] + end + + result = orchestrator.execute_plan + [result, tasks] +end + +def print_train(label, result, tasks) + puts label + tasks.each do |station, task| + task_result = result.results[task.id] + status = if task_result.nil? + "CANCELED (never left the yard)" + elsif task_result.successful? + "green" + elsif task_result.canceled? + "CANCELED" + else + "RED - #{task_result.failure.message}" + end + puts format(" %-9s %s", station, status) + end + puts format(" train status: %s", result.status) + puts +end + +healthy, healthy_tasks = run_train(canary_healthy: true) +print_train("monday's deploy:", healthy, healthy_tasks) + +broken, broken_tasks = run_train(canary_healthy: false) +print_train("friday's deploy:", broken, broken_tasks) + +puts "friday's verdict is precise: the train is :partial_failure (a gate" +puts "went RED), and the cars behind it are CANCELED, not silently" +puts "skipped. failure outranks cancellation in the status - the headline" +puts "is WHY the train stopped, the manifest shows what never shipped." From 09534d2d8f41e0d3250f4968a050170bf899b6b3 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 7 Jul 2026 01:27:01 +0000 Subject: [PATCH 16/36] docs: perf diff example (tenderlove round 6) Before/after plan measurement with a noise floor and critical-path qualification: a PR that speeds up slack time while regressing the path gets an exit-code veto. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01FeZLdZ3M4q4cho4DZPfSkF --- docs/perspectives/round-6/03-tenderlove.md | 58 +++++++++++++ examples/perf_diff.rb | 98 ++++++++++++++++++++++ 2 files changed, 156 insertions(+) create mode 100644 docs/perspectives/round-6/03-tenderlove.md create mode 100644 examples/perf_diff.rb diff --git a/docs/perspectives/round-6/03-tenderlove.md b/docs/perspectives/round-6/03-tenderlove.md new file mode 100644 index 0000000..2cbb687 --- /dev/null +++ b/docs/perspectives/round-6/03-tenderlove.md @@ -0,0 +1,58 @@ +# Round 6 field notes — Aaron Patterson diffs the runs + +*Built: `examples/perf_diff.rb` — the plan measured before and after +"the PR," per-task deltas with a noise floor, and the one qualifier +that decides everything: is the regression on the critical path?* + +## What I built and why + +Every perf regression I've ever chased shipped inside a PR that made +something *else* faster. The diff tool models that exact crime scene: +the PR halves `reprice:catalog` (real win, -50ms!) and quietly makes +`fetch:prices` slower (+59ms). A naive per-task diff calls that a +wash. The wall clock disagrees: + +``` +reprice:catalog -50ms faster +fetch:prices +59ms SLOWER + ON CRITICAL PATH +wall clock: 231ms -> 241ms (+10ms) +VERDICT: don't ship. +``` + +The repricing win lands on slack time; the fetch regression lands on +the path that bounds the plan. Users get the regression and none of +the win. That asymmetry is the single most misunderstood fact in +performance work, and this tool prints it with an exit code so CI can +refuse the PR before a human has to win the argument. + +## Craftsman's notes + +- **The noise floor is not optional.** 15ms here; without it, every + scheduler wobble becomes a "regression" and the tool trains people + to ignore it. A perf gate that cries wolf is worse than none — + calibrate the floor to your variance, then trust it. +- The critical-path check reuses round 5's fifteen-line walk, now over + `graph[:order]` instead of my hand-rolled traversal — the toposort + ask, cashed within one round. Third round in a row where the ask → + ship → use loop closed inside a single iteration; I've stopped being + surprised and started being spoiled. +- Verdict design: off-path regressions print lowercase ("slower + (off-path)") and don't block. They're real, they're recorded, and + they're *not the headline*. Alert fatigue is a perf bug in your + process. + +## Where this goes + +Baseline durations belong in a JSON artifact committed per-release, so +the diff runs against recorded history instead of a same-process rerun +— Perham's journal already stores per-task durations, which is the +natural source. Wire journal → baseline → this diff and you have +continuous plan-performance regression testing for the price of a +cron job. + +## Verdict + +Gantt: where time went. Knee: how many lanes. Path: which task +matters. Diff: **did the PR make it worse.** The performance suite is +complete enough that I'd hand it to an ops team — four tools, ~400 +lines total, all standing on two hooks and one accessor. diff --git a/examples/perf_diff.rb b/examples/perf_diff.rb new file mode 100644 index 0000000..8dd8ef9 --- /dev/null +++ b/examples/perf_diff.rb @@ -0,0 +1,98 @@ +# frozen_string_literal: true + +# The Perf Diff: run the plan before and after a change, diff per-task +# durations, and flag regressions - with the one qualifier that decides +# whether anyone should care: is the regressed task ON the critical +# path? Off-path regressions are trivia; on-path regressions are the +# release note nobody wrote. +# +# bundle exec ruby examples/perf_diff.rb +# +# Runs offline; the "change" speeds one task up and quietly breaks +# another, as changes do. + +require_relative "../lib/agentic" + +BASELINE = { + "fetch:prices" => {sleep: 0.10, deps: []}, + "fetch:inventory" => {sleep: 0.06, deps: []}, + "reprice:catalog" => {sleep: 0.08, deps: ["fetch:prices", "fetch:inventory"]}, + "index:search" => {sleep: 0.05, deps: ["reprice:catalog"]}, + "warm:cache" => {sleep: 0.04, deps: ["reprice:catalog"]} +}.freeze + +# The optimization sped up repricing... and the same PR made +# fetch:prices slower. Ship it? Let's find out. +AFTER_THE_PR = BASELINE.merge( + "reprice:catalog" => BASELINE["reprice:catalog"].merge(sleep: 0.03), + "fetch:prices" => BASELINE["fetch:prices"].merge(sleep: 0.16) +).freeze + +def measure(work) + durations = {} + orchestrator = Agentic::PlanOrchestrator.new( + concurrency_limit: 4, + lifecycle_hooks: { + after_task_success: ->(task_id:, task:, result:, duration:) { durations[task.description] = duration } + } + ) + tasks = {} + work.each do |name, spec| + tasks[name] = Agentic::Task.new(description: name, + agent_spec: {"name" => name, "instructions" => "work"}, payload: spec[:sleep]) + orchestrator.add_task(tasks[name], spec[:deps].map { |d| tasks.fetch(d) }, + agent: ->(t) { sleep(t.payload) || :ok }) + end + result = orchestrator.execute_plan + [durations, result.execution_time, orchestrator.graph] +end + +def critical_path_names(graph, durations) + names = graph[:tasks].transform_values(&:description) + memo = {} + walk = lambda do |task_id| + memo[task_id] ||= begin + best = graph[:dependencies][task_id].map { |d| walk.call(d) }.max_by { |p| p[:cost] } || {cost: 0.0, path: []} + {cost: best[:cost] + durations[names[task_id]], path: best[:path] + [names[task_id]]} + end + end + graph[:order].map { |id| walk.call(id) }.max_by { |p| p[:cost] }[:path] +end + +NOISE_MS = 15 + +before, wall_before, = measure(BASELINE) +after, wall_after, graph = measure(AFTER_THE_PR) +path_after = critical_path_names(graph, after) + +puts "PERF DIFF (noise floor #{NOISE_MS}ms)" +puts +puts format(" %-18s %9s %9s %9s %s", "task", "before", "after", "delta", "") +regressions = [] +BASELINE.each_key do |name| + delta_ms = (after[name] - before[name]) * 1000 + marker = + if delta_ms.abs < NOISE_MS then "" + elsif delta_ms.negative? then "faster" + else + on_path = path_after.include?(name) + regressions << {name: name, on_path: on_path} + on_path ? "SLOWER + ON CRITICAL PATH" : "slower (off-path)" + end + puts format(" %-18s %7dms %7dms %+8dms %s", + name, before[name] * 1000, after[name] * 1000, delta_ms, marker) +end + +puts +puts format(" wall clock: %dms -> %dms (%+dms)", + wall_before * 1000, wall_after * 1000, (wall_after - wall_before) * 1000) +puts +blocking = regressions.select { |r| r[:on_path] } +if blocking.any? + puts " VERDICT: don't ship. #{blocking.map { |r| r[:name] }.join(", ")} regressed" + puts " on the critical path - the repricing win is real and the users" + puts " will never feel it, because the wall clock got worse anyway." + exit 1 +else + puts " VERDICT: ship it." +end From 43de83359a68d96d418a9ed0146484177db538b3 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 7 Jul 2026 01:28:09 +0000 Subject: [PATCH 17/36] docs: plan round-trip example (fxn round 6) Graph to JSON to fresh orchestrator and back, with an isomorphism check: identity travels as description, structure as labeled edges, behavior deliberately does not serialize. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01FeZLdZ3M4q4cho4DZPfSkF --- docs/perspectives/round-6/04-fxn.md | 56 ++++++++++++++++++ examples/plan_roundtrip.rb | 91 +++++++++++++++++++++++++++++ 2 files changed, 147 insertions(+) create mode 100644 docs/perspectives/round-6/04-fxn.md create mode 100644 examples/plan_roundtrip.rb diff --git a/docs/perspectives/round-6/04-fxn.md b/docs/perspectives/round-6/04-fxn.md new file mode 100644 index 0000000..97d09d8 --- /dev/null +++ b/docs/perspectives/round-6/04-fxn.md @@ -0,0 +1,56 @@ +# Round 6 field notes — Xavier Noria closes the round trip + +*Built: `examples/plan_roundtrip.rb` — graph → JSON → fresh +orchestrator → graph, with an isomorphism check proving nothing was +lost in either direction.* + +## What I built and why + +A projection you can only read is a report; a projection you can +*invert* is a format. Round 5's diagrammer read the graph outward +(graph → Mermaid); this round closes the loop: serialize the topology +to JSON, rebuild a brand-new orchestrator from that JSON, and compare +shapes. The verdict line is the deliverable: + +``` +round trip is faithful: 3 edges, labels intact, +topological order preserved (gather -> check -> weave -> ship) +``` + +With this, plans become *artifacts*: you can commit them, diff them in +review, hand them between processes, build them in a UI and execute +them in a worker. The CLI's existing `--save plan.json` stores the +planner's *task list*; this wire format stores the *topology* — which +is the part that round 3's piping and round 4's `needs:` made worth +preserving. + +## The two decisions that make the format sound + +1. **Ids do not travel.** Task ids are per-process UUIDs; a wire + format that ships them is shipping garbage that will collide or + mislead. Identity travels as description, structure as edges — the + same discipline as Perham's journal keys. The isomorphism check + accordingly compares *shapes* (names and labeled edges), never ids. +2. **`graph[:edges]` is the serialization surface.** Because round 6's + release pre-merges positional and named dependencies into labeled + edge records, serialize + deserialize are ~15 lines each with no + special cases. My round-5 diagrammer had to dedupe needs against + dependencies by hand; that logic simply no longer exists anywhere + in this file. When a release deletes code from *examples that + haven't been written yet*, the API changed at the right layer. + +## The caveat, stated plainly + +The wire format carries topology, not behavior: agents and payloads +are lambdas and live objects, which do not serialize (and must not — +`Marshal`ing closures is how gems end up in CVE databases). A rebuilt +plan needs its agents re-attached by name, exactly as the CLI +re-resolves agents from specs. Structure travels; capability is +re-granted at the destination. That is a *feature* with security +posture, not a gap. + +## Verdict + +The graph now survives translation in both directions with a proof +attached. Plans-as-data was always the promise of the plan-and-execute +architecture; as of this round the data part is honest. diff --git a/examples/plan_roundtrip.rb b/examples/plan_roundtrip.rb new file mode 100644 index 0000000..163e309 --- /dev/null +++ b/examples/plan_roundtrip.rb @@ -0,0 +1,91 @@ +# frozen_string_literal: true + +# The Round Trip: serialize a plan's graph to JSON, rebuild a fresh +# orchestrator from the JSON, and prove the rebuilt topology is +# isomorphic to the original - same shape, same labels, new ids. A +# projection you can't invert is a projection you can't trust with +# your plans. +# +# bundle exec ruby examples/plan_roundtrip.rb +# +# Runs offline; prints the wire format and the verdict. + +require_relative "../lib/agentic" +require "json" + +def step(name) + Agentic::Task.new(description: name, agent_spec: {"name" => name, "instructions" => "work"}) +end + +# --- an original plan with every edge flavor --------------------------------- +original = Agentic::PlanOrchestrator.new +gather = step("gather") +check = step("check") +weave = step("weave") +ship = step("ship") + +original.add_task(gather) +original.add_task(check) +original.add_task(weave, [check], needs: {threads: gather}) +original.add_task(ship, [weave]) + +# --- serialize: graph -> wire format (ids replaced by descriptions) --------- +def serialize(graph) + names = graph[:tasks].transform_values(&:description) + { + "tasks" => graph[:order].map { |id| names[id] }, + "edges" => graph[:edges].map { |e| + {"from" => names[e[:from]], "to" => names[e[:to]], "label" => e[:label]&.to_s} + } + } +end + +# --- deserialize: wire format -> a fresh orchestrator ------------------------ +def deserialize(data) + orchestrator = Agentic::PlanOrchestrator.new + tasks = data["tasks"].to_h { |name| [name, step(name)] } + + data["tasks"].each do |name| + edges_in = data["edges"].select { |e| e["to"] == name } + plain = edges_in.reject { |e| e["label"] }.map { |e| tasks.fetch(e["from"]) } + named = edges_in.select { |e| e["label"] } + .to_h { |e| [e["label"].to_sym, tasks.fetch(e["from"])] } + + orchestrator.add_task(tasks[name], plain, needs: named.empty? ? nil : named) + end + orchestrator +end + +wire = JSON.pretty_generate(serialize(original.graph)) +rebuilt = deserialize(JSON.parse(wire)) + +puts "THE WIRE FORMAT" +puts wire.gsub(/^/, " ") +puts + +# --- the isomorphism check: compare shapes, not ids -------------------------- +def shape(graph) + names = graph[:tasks].transform_values(&:description) + { + order: graph[:order].map { |id| names[id] }, + edges: graph[:edges].map { |e| [names[e[:from]], names[e[:to]], e[:label]] }.sort_by(&:to_s) + } +end + +before = shape(original.graph) +after = shape(rebuilt.graph) + +puts "THE VERDICT" +if before == after + puts " round trip is faithful: #{before[:edges].size} edges, labels intact," + puts " topological order preserved (#{after[:order].join(" -> ")})" +else + puts " DRIFT DETECTED:" + puts " before: #{before.inspect}" + puts " after: #{after.inspect}" + exit 1 +end +puts +puts "task ids are per-process and correctly absent from the wire format -" +puts "identity travels as description, structure travels as edges, and" +puts "needs: labels survive because graph[:edges] carries them." From 29c05fe387f78ffabe7ae15ce281b22674f7f5e7 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 7 Jul 2026 01:29:19 +0000 Subject: [PATCH 18/36] docs: quota keeper example (ioquatix round 6) The same twenty requests through a concurrency ceiling (done in 61ms) and the new windowed quota (601ms): different physics, one class, admission charts as proof. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01FeZLdZ3M4q4cho4DZPfSkF --- docs/perspectives/round-6/05-ioquatix.md | 53 ++++++++++++++++++++ examples/quota_keeper.rb | 64 ++++++++++++++++++++++++ 2 files changed, 117 insertions(+) create mode 100644 docs/perspectives/round-6/05-ioquatix.md create mode 100644 examples/quota_keeper.rb diff --git a/docs/perspectives/round-6/05-ioquatix.md b/docs/perspectives/round-6/05-ioquatix.md new file mode 100644 index 0000000..12e441d --- /dev/null +++ b/docs/perspectives/round-6/05-ioquatix.md @@ -0,0 +1,53 @@ +# Round 6 field notes — Samuel Williams keeps the quota + +*Built: `examples/quota_keeper.rb` — the same twenty requests through a +concurrency ceiling and through the new windowed quota; the admission +charts show they are different physics.* + +## What I built and why + +Round 5 I said concurrency ceilings model connection limits and +windowed quotas model billing, and that both belong. The windowed mode +shipped (`RateLimit.new(5, per: 0.2)`), so this round the two laws +face the same twenty requests: + +``` +concurrency 3: 0-200ms #################### 20 (done by 61ms) +window 5/200ms: 0-200ms ##### 5 + 200-400ms ##### 5 + 400-600ms ##### 5 + 600-800ms ##### 5 (done by 601ms) +``` + +Same workload, **61ms versus 601ms** — a 10x difference dictated +entirely by which law you chose. Under a concurrency ceiling, +finishing early frees a slot, so fast calls drain the queue at IO +speed. Under a window, finishing early buys *nothing*: five per +period is five per period. Anyone who has ever "optimized" their API +calls and watched throughput not move has met a window while modeling +a ceiling — the chart pair is that lesson in one screen. + +## Implementation review (I read the shipped code first, as always) + +- The windowed algorithm is the honest simple one: prune stamps older + than the window, admit if under ceiling, else `sleep` until the + oldest stamp exits — and that sleep suspends only the *acquiring + fiber*, so unrelated reactor work proceeds. Fiber-friendly waiting + is the whole reason this can live in-process instead of in Redis. +- Stamps record at *admission*, making this a sliding-window-of-starts + — the same law OpenAI and Anthropic quota by. A leaky-bucket variant + smooths differently; the admission-stamp choice is the right default + because it matches what providers measure. +- One thing to document: windowed mode intentionally does not bound + *concurrency* (all five of a window's calls may be in flight + together). Production wants both laws at once — a window for quota + and a ceiling for sockets - which today means nesting two limiters. + `RateLimit.new(3, and_per_window: [30, 60])`... no. Two objects, + composed. The primitive is right; the composition is the user's + sentence to write. + +## Verdict + +Two laws, one class, an honest chart apiece. The rate-limiting story +that started as a crowbar in round 4 is now a taxonomy with +measurements — which is what infrastructure maturity looks like. diff --git a/examples/quota_keeper.rb b/examples/quota_keeper.rb new file mode 100644 index 0000000..16d9deb --- /dev/null +++ b/examples/quota_keeper.rb @@ -0,0 +1,64 @@ +# frozen_string_literal: true + +# The Quota Keeper: the same 20 requests through two different laws. +# A concurrency ceiling ("3 in flight") models connection limits; a +# windowed quota ("5 per 200ms") models what providers actually bill. +# They are different physics, and the admission timeline proves it. +# +# bundle exec ruby examples/quota_keeper.rb +# +# Runs offline; calls are 10ms of simulated IO. + +require_relative "../lib/agentic" +require "async" + +REQUESTS = 20 +CALL_TIME = 0.01 + +def fire_through(limit) + admissions = [] + started = Process.clock_gettime(Process::CLOCK_MONOTONIC) + + Sync do + REQUESTS.times.map { + Async do + limit.acquire do + admissions << Process.clock_gettime(Process::CLOCK_MONOTONIC) - started + sleep(CALL_TIME) + end + end + }.each(&:wait) + end + + admissions.sort +end + +def admission_chart(admissions, bucket = 0.2) + buckets = admissions.group_by { |t| (t / bucket).floor } + (0..admissions.max / bucket).map { |i| + format(" %3d-%3dms %-22s %d", + i * bucket * 1000, (i + 1) * bucket * 1000, "#" * (buckets[i]&.size || 0), buckets[i]&.size || 0) + }.join("\n") +end + +puts "QUOTA KEEPER: #{REQUESTS} requests, #{(CALL_TIME * 1000).round}ms each, fired simultaneously" +puts + +concurrent = fire_through(Agentic::RateLimit.new(3)) +puts " law 1 - concurrency ceiling (3 in flight):" +puts admission_chart(concurrent) +puts format(" all admitted by %dms - completion frees a slot, so short", concurrent.last * 1000) +puts " calls drain the queue as fast as they finish" +puts + +windowed = fire_through(Agentic::RateLimit.new(5, per: 0.2)) +puts " law 2 - windowed quota (5 per 200ms):" +puts admission_chart(windowed) +puts format(" last admitted at %dms - finishing early buys NOTHING;", windowed.last * 1000) +puts " the window admits five per period no matter how quick the calls" +puts + +puts "same requests, ~#{(concurrent.last * 1000).round}ms versus ~#{(windowed.last * 1000).round}ms: pick the law your" +puts "provider actually enforces. connection pools are ceilings; billed" +puts "quotas are windows; production APIs are usually both at once -" +puts "which is why RateLimit lets you hold one of each." From 76100400fb29173ce047e854f1883d53b9e6b572 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 7 Jul 2026 01:30:32 +0000 Subject: [PATCH 19/36] docs: rule prober example (Jeremy Evans round 6) Metamorphic audit of structured rules' field declarations: perturbing undeclared fields must never flip a verdict. A seeded lying rule is caught in 71 of 300 trials, with the UX harm explained. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01FeZLdZ3M4q4cho4DZPfSkF --- docs/perspectives/round-6/06-jeremyevans.md | 59 +++++++++++ examples/rule_prober.rb | 104 ++++++++++++++++++++ 2 files changed, 163 insertions(+) create mode 100644 docs/perspectives/round-6/06-jeremyevans.md create mode 100644 examples/rule_prober.rb diff --git a/docs/perspectives/round-6/06-jeremyevans.md b/docs/perspectives/round-6/06-jeremyevans.md new file mode 100644 index 0000000..d7e0191 --- /dev/null +++ b/docs/perspectives/round-6/06-jeremyevans.md @@ -0,0 +1,59 @@ +# Round 6 field notes — Jeremy Evans probes the rules + +*Built: `examples/rule_prober.rb` — now that rules declare their +fields, that declaration is a testable claim. One seeded rule lies +about what it reads; the prober catches it in 71 of 300 trials.* + +## What I built and why + +In round 5 I wrote that rule lambdas were opaque to fuzzing — "an +acceptable trade." The structured-rules release quietly changed the +trade: `fields:` is a *claim about the lambda's data dependencies*, +and claims can be audited. The probe is metamorphic testing at its +simplest: generate a conforming application, record the rule's +verdict, perturb a field the rule does NOT declare, and check the +verdict didn't move. Three hundred trials per rule, seeded: + +``` +affordability: honest +subprime_needs_cosigner: honest +jumbo_screening: LYING - declares [:amount] but its verdict + flipped when :score changed (71/300 trials) +``` + +`jumbo_screening` reads `:score` off the books. And the harm is +concrete, not aesthetic: Piotr's 422 renderer highlights *declared* +fields, so this rule sends a user to fix `:amount` while the actual +problem — their credit score — sits unhighlighted on the form. A lying +declaration turns a helpful error into a misdirection. That's a +correctness bug in the UX, planted two layers away from any UI code. + +## Methodology notes + +- Perturbations draw from the same conforming generator as the + originals, so a flip can never be blamed on invalid data — the probe + isolates exactly one variable. Fuzzing 101, still worth stating. +- 71/300 flips, not 300/300: the lie only surfaces when + `amount > 400k` AND the score perturbation crosses 700. Partial + sensitivity is exactly why eyeballing rules doesn't find this — + probabilistic bugs need volume, volume needs automation, automation + needs a seed. +- The exit code makes it CI: `rule_prober` alongside the contract + fuzzer (round 3) and the README verifier (round 4). The honesty + suite now covers types, promises, and — new — *declarations*. + +## The observation about API design + +Nobody designed `fields:` as a testability feature. It shipped (round +6 release) as UI plumbing — "let the 422 highlight widgets." But any +declaration, once machine-readable, becomes a specification you can +check the implementation against. This is the recurring miracle of +typed metadata: **write down what you believe and the tests write +themselves.** It's why I keep asking every round for more things to +be data. + +## Verdict + +Three rules audited, one liar caught, one seed to reproduce it. The +declarations are honest now — and more importantly, they're *kept* +honest by a forty-line program any CI can run. diff --git a/examples/rule_prober.rb b/examples/rule_prober.rb new file mode 100644 index 0000000..3f71ee0 --- /dev/null +++ b/examples/rule_prober.rb @@ -0,0 +1,104 @@ +# frozen_string_literal: true + +# The Rule Prober: structured rules declare which fields they read - +# so now that claim can be AUDITED. For each rule, perturb fields it +# does NOT declare; if the verdict flips, the rule is reading fields +# off the books. One of the rules below lies. The prober finds it. +# +# bundle exec ruby examples/rule_prober.rb [seed] +# +# Runs offline and deterministically. + +require_relative "../lib/agentic" + +seed = (ARGV.first || 20260707).to_i +rng = Random.new(seed) + +SPEC = Agentic::CapabilitySpecification.new( + name: "approve_loan", + description: "Approve a loan application", + version: "1.0.0", + inputs: { + amount: {type: "number", required: true, min: 1}, + income: {type: "number", required: true, min: 0}, + score: {type: "number", required: true, min: 300, max: 850}, + cosigned: {type: "boolean", required: true} + }, + rules: { + affordability: { + message: "amount may not exceed 5x income", + fields: [:amount, :income], + check: ->(i) { i[:amount] <= i[:income] * 5 } + }, + subprime_needs_cosigner: { + message: "scores under 600 require a cosigner", + fields: [:score, :cosigned], + check: ->(i) { i[:score] >= 600 || i[:cosigned] } + }, + # The liar: declares [:amount] but secretly reads :score too + jumbo_screening: { + message: "amounts over 400k get extra screening", + fields: [:amount], + check: ->(i) { i[:amount] <= 400_000 || i[:score] > 700 } + } + } +) + +# Generate a conforming application +def sample_application(rng) + { + amount: rng.rand(10_000..900_000), + income: rng.rand(30_000..250_000), + score: rng.rand(300..850), + cosigned: [true, false].sample(random: rng) + } +end + +# Perturb one field to another conforming value +def perturb(application, field, rng) + fresh = sample_application(rng) + application.merge(field => fresh[field]) +end + +TRIALS = 300 +findings = Hash.new { |h, k| h[k] = [] } + +SPEC.rules.each do |rule_id, definition| + undeclared = SPEC.inputs.keys - definition[:fields] + + TRIALS.times do + application = sample_application(rng) + verdict = definition[:check].call(application) + + undeclared.each do |field| + mutated = perturb(application, field, rng) + next if mutated == application + + if definition[:check].call(mutated) != verdict + findings[rule_id] << field + end + end + end +end + +puts "RULE PROBER (seed #{seed}, #{TRIALS} trials per rule)" +puts +SPEC.rules.each do |rule_id, definition| + flipped = findings[rule_id].tally + if flipped.empty? + puts " #{rule_id}: honest - declares #{definition[:fields].inspect}, " \ + "and no undeclared field ever changed its verdict" + else + puts " #{rule_id}: LYING - declares #{definition[:fields].inspect} but its verdict" + flipped.each do |field, count| + puts " flipped when :#{field} changed (#{count} of #{TRIALS} trials)" + end + end +end + +puts +puts "why it matters: Piotr's 422 renderer highlights the DECLARED fields." +puts "a rule that secretly reads :score sends users to fix :amount while" +puts "the real problem sits unhighlighted. field declarations are now" +puts "testable claims - so test them." +exit findings.empty? ? 0 : 1 From cc12bdf35ffc0a4f2db482647eff3cbdf7b83630 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 7 Jul 2026 01:31:44 +0000 Subject: [PATCH 20/36] docs: API reference generator example (solnic round 6) Markdown reference docs emitted from the registry's contracts - types, bounds, enums, and structured policies with their fields. One declaration now validates, rejects, explains, documents, and gets audited. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01FeZLdZ3M4q4cho4DZPfSkF --- docs/perspectives/round-6/07-solnic.md | 61 ++++++++++++++ examples/api_reference.rb | 111 +++++++++++++++++++++++++ 2 files changed, 172 insertions(+) create mode 100644 docs/perspectives/round-6/07-solnic.md create mode 100644 examples/api_reference.rb diff --git a/docs/perspectives/round-6/07-solnic.md b/docs/perspectives/round-6/07-solnic.md new file mode 100644 index 0000000..be78012 --- /dev/null +++ b/docs/perspectives/round-6/07-solnic.md @@ -0,0 +1,61 @@ +# Round 6 field notes — Piotr Solnica generates the API reference + +*Built: `examples/api_reference.rb` — walk the registry, emit markdown +reference docs from the same contracts that validate every call.* + +## What I built and why + +The endgame of contract-first design was always this: the contract is +the validator, the error renderer, *and* the documentation — one +artifact, three projections. The generator walks the registry and +emits a reference page per capability: + +``` +| `amount_cents` | number | yes | >= 1; <= 10000000 | Amount in cents | + +### Policies +- **no_self_transfer** (checks `from_account`, `to_account`): + source and destination must differ +``` + +Every column is read from the declaration: types, requiredness, enums, +bounds, non-empty, and — thanks to this round's structured rules — the +policies with the fields they check. API docs that drift from the +validator are the industry's most-shipped lie; these *cannot* drift, +because the sentence that documents the bound is generated from the +object that enforces it. + +## The arc, made explicit + +Trace `amount_cents: {min: 1}` through the rounds: round 2, types were +decorative. Round 4, `min:` rejects bad input. Round 5, the rejection +carries the bound (`expectations`). Round 6, the bound documents +itself in a table AND Jeremy's prober audits the rules' field claims. +**One declaration, five behaviors** — validate, reject, explain, +document, audit. That multiplication is the entire argument for +schema-as-data over validation-as-code; I have been making it in +conference talks for a decade and this gem now makes it in 100 lines +of examples. + +## Notes + +- `description:` on input declarations was already in the + specification shape (round 2's capabilities used it) but nothing + consumed it until this table. Metadata survives waiting for its + consumer — another reason to prefer declarations over code. +- The near-term upgrade is emitting JSON Schema / OpenAPI instead of + markdown — the declarations map 1:1 (`enum`→`enum`, `min`→`minimum`, + `non_empty`→`minLength/minItems`). At that point Agentic capability + contracts become importable into every API toolchain on earth. An + afternoon, for whoever wants it. +- What the reference can't document: prose-form rules (no fields, no + id — they render as their description only) and implementation + behavior beyond the boundary. Correct limits, both; documentation + generators should document claims, not guess at code. + +## Verdict + +Contracts now validate, explain, document, and get audited — four +consumers of one declaration, none of which can disagree with the +others. That's the property "single source of truth" was always +supposed to mean. diff --git a/examples/api_reference.rb b/examples/api_reference.rb new file mode 100644 index 0000000..f232d2d --- /dev/null +++ b/examples/api_reference.rb @@ -0,0 +1,111 @@ +# frozen_string_literal: true + +# The API Reference Generator: walk the registry, emit reference docs +# for every capability - types, enums, bounds, policies - straight from +# the contracts that VALIDATE the calls. Documentation that enforces +# itself cannot lie about what it accepts. +# +# bundle exec ruby examples/api_reference.rb +# +# Runs offline; prints markdown. + +require_relative "../lib/agentic" + +registry = Agentic::AgentCapabilityRegistry.instance + +# Two capabilities as the "app": a transfer and a payout +transfer = Agentic::CapabilitySpecification.new( + name: "transfer_funds", + description: "Move money between accounts", + version: "1.2.0", + inputs: { + from_account: {type: "string", required: true, non_empty: true, description: "Source account id"}, + to_account: {type: "string", required: true, non_empty: true, description: "Destination account id"}, + amount_cents: {type: "number", required: true, min: 1, max: 10_000_000, description: "Amount in cents"}, + memo: {type: "string", description: "Optional statement memo"} + }, + outputs: { + transfer_id: {type: "string", required: true}, + settled: {type: "boolean", required: true} + }, + rules: { + no_self_transfer: { + message: "source and destination must differ", + fields: [:from_account, :to_account], + check: ->(i) { i[:from_account] != i[:to_account] } + } + } +) + +payout = Agentic::CapabilitySpecification.new( + name: "schedule_payout", + description: "Schedule a payout to a bank account", + version: "2.0.0", + inputs: { + amount_cents: {type: "number", required: true, min: 100}, + speed: {type: "string", required: true, enum: %w[standard instant]}, + currency: {type: "string", required: true, enum: %w[usd eur gbp]} + }, + outputs: {payout_id: {type: "string", required: true}}, + rules: { + instant_is_domestic: { + message: "instant payouts support usd only", + fields: [:speed, :currency], + check: ->(i) { i[:speed] != "instant" || i[:currency] == "usd" } + } + } +) + +[transfer, payout].each do |spec| + Agentic.register_capability(spec, Agentic::CapabilityProvider.new( + capability: spec, implementation: ->(_i) { {} } + )) +end + +# --- the generator: registry in, markdown out -------------------------------- +def constraint_notes(declaration) + notes = [] + notes << "one of: #{declaration[:enum].join(", ")}" if declaration[:enum] + notes << ">= #{declaration[:min]}" if declaration[:min] + notes << "<= #{declaration[:max]}" if declaration[:max] + notes << "non-empty" if declaration[:non_empty] + notes.join("; ") +end + +def field_table(declared) + rows = declared.map { |name, decl| + required = decl[:required] ? "yes" : "no" + format("| `%s` | %s | %s | %s | %s |", + name, decl[:type], required, constraint_notes(decl), decl[:description] || "") + } + ["| Field | Type | Required | Constraints | Description |", + "|-------|------|----------|-------------|-------------|"] + rows +end + +def reference_for(spec) + doc = ["## `#{spec.name}` v#{spec.version}", "", spec.description, ""] + doc << "### Inputs" + doc += field_table(spec.inputs) + unless spec.rules.empty? + doc << "" + doc << "### Policies" + spec.rules.each do |rule_id, rule| + doc << "- **#{rule_id}** (checks #{rule[:fields].map { |f| "`#{f}`" }.join(", ")}): #{rule[:message]}" + end + end + unless spec.outputs.empty? + doc << "" + doc << "### Outputs" + doc += field_table(spec.outputs) + end + doc.join("\n") +end + +puts "# API Reference" +puts +puts "_Generated from the same contracts that validate every call._" +puts +%w[transfer_funds schedule_payout].each do |name| + puts reference_for(registry.get(name)) + puts +end From 3f069fbcf1c123005fe7203f354d8a7bd0f581b1 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 7 Jul 2026 01:32:57 +0000 Subject: [PATCH 21/36] docs: jitter shootout example (mperham round 6) None vs equal vs full jitter on one seeded scoreboard: peak retry herd 40 -> 19 -> 13. Field notes give the operational guidance for choosing a tier. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01FeZLdZ3M4q4cho4DZPfSkF --- docs/perspectives/round-6/08-mperham.md | 56 +++++++++++++++++ examples/jitter_shootout.rb | 84 +++++++++++++++++++++++++ 2 files changed, 140 insertions(+) create mode 100644 docs/perspectives/round-6/08-mperham.md create mode 100644 examples/jitter_shootout.rb diff --git a/docs/perspectives/round-6/08-mperham.md b/docs/perspectives/round-6/08-mperham.md new file mode 100644 index 0000000..afa144b --- /dev/null +++ b/docs/perspectives/round-6/08-mperham.md @@ -0,0 +1,56 @@ +# Round 6 field notes — Mike Perham judges the jitter shootout + +*Built: `examples/jitter_shootout.rb` — none vs equal (the default) vs +full (this round's release), same forty synchronized workers, three +histograms, one scoreboard.* + +## What I built and why + +Round 5's stampede sim argued for the jitter *default*; this round the +`:full` tier shipped, so the shootout puts all three modes on one +seeded scoreboard: + +``` +none: peak 40 of 40 spread 2ms +equal +/-25%: peak 19 of 40 spread 75ms +full [0,d]: peak 13 of 40 spread 152ms +``` + +No jitter is a synchronized herd — all forty in one 25ms bucket, +which is a DDoS you scheduled against yourself. Equal jitter (the +default) halves the peak. Full jitter — retries drawn uniformly from +`[0, delay]` — cuts it to a third and doubles the spread, at the cost +of *punctuality*: some workers retry almost immediately, one waits +nearly the full backoff. That trade is exactly right for the +recovering-upstream case: **when the service is already hurting, +"together" is the only wrong arrival time.** This matches the AWS +Architecture Blog's classic analysis, and now the gem's histogram +matches the literature's math, reproducibly, with a seed. + +## Operational guidance, straight from the table + +- Default (equal) is correct for most fleets: bounded delay variance + keeps p99 retry latency predictable, peak halves for free. +- Switch to `:full` when the retry *target* is the bottleneck — rate + limits, brownouts, thundering-herd-prone startup paths. You're + trading your own latency tail for the upstream's survival, which is + the right trade because a dead upstream has infinite latency. +- `none` is for tests asserting exact delays, and nothing else. It's + opt-in now, which is where footguns belong: in the drawer, labeled. + +## Notes + +- The three-mode comparison needed zero framework hooks — arrivals + recorded in the agent, seeded via `srand` since the policy uses + `Kernel#rand`. Policy knobs that honor global seeding are testable + policy knobs; whoever wires custom RNG injection later should keep + that property. +- Peak-per-bucket is the metric because upstreams die of *instantaneous* + concurrency, not totals. Forty retries over 150ms is fine; forty in + 2ms is an incident. Measure what kills you. + +## Verdict + +Three modes, one seed, one scoreboard: 40 → 19 → 13. The default +protects everyone, the `:full` tier protects the wounded, and the +histogram means nobody has to take my word for any of it anymore. diff --git a/examples/jitter_shootout.rb b/examples/jitter_shootout.rb new file mode 100644 index 0000000..55467fc --- /dev/null +++ b/examples/jitter_shootout.rb @@ -0,0 +1,84 @@ +# frozen_string_literal: true + +# The Jitter Shootout: none vs equal (+/-25%, the default) vs full +# (uniform over [0, delay], new this round) - same forty workers, same +# synchronized failure, three retry-arrival histograms. Pick your +# herd size with your eyes open. +# +# bundle exec ruby examples/jitter_shootout.rb [seed] +# +# Runs offline and deterministically. + +require_relative "../lib/agentic" + +Agentic.logger.level = :fatal + +WORKERS = 40 +BACKOFF = 0.15 +BUCKET_MS = 25 + +def run_mode(jitter, seed) + srand(seed) + started = Process.clock_gettime(Process::CLOCK_MONOTONIC) + arrivals = [] + attempts = Hash.new(0) + + orchestrator = Agentic::PlanOrchestrator.new( + concurrency_limit: WORKERS, + retry_policy: { + max_retries: 2, + retryable_errors: ["RuntimeError"], + backoff_strategy: :constant, + backoff_constant: BACKOFF, + backoff_jitter: jitter + } + ) + + WORKERS.times do |i| + orchestrator.add_task(Agentic::Task.new( + description: "worker-#{i}", + agent_spec: {"name" => "worker", "instructions" => "call"} + ), agent: ->(t) { + attempts[t.description] += 1 + raise "hiccup" if attempts[t.description] == 1 + + arrivals << Process.clock_gettime(Process::CLOCK_MONOTONIC) - started + :ok + }) + end + orchestrator.execute_plan + arrivals +end + +def report(label, arrivals) + buckets = arrivals.group_by { |t| (t * 1000 / BUCKET_MS).floor } + peak = buckets.values.map(&:size).max + spread = ((arrivals.max - arrivals.min) * 1000).round + + puts " #{label}:" + buckets.sort.each do |bucket, hits| + puts format(" %4d-%4dms %-40s %d", bucket * BUCKET_MS, (bucket + 1) * BUCKET_MS, + "#" * hits.size, hits.size) + end + puts format(" peak %d workers per bucket, spread %dms", peak, spread) + puts + peak +end + +seed = (ARGV.first || 20260707).to_i +puts "JITTER SHOOTOUT: #{WORKERS} workers, synchronized failure, " \ + "#{(BACKOFF * 1000).round}ms base backoff (seed #{seed})" +puts + +peaks = { + "none (jitter: false)" => run_mode(false, seed), + "equal +/-25% (the default)" => run_mode(true, seed), + "full [0, delay] (jitter: :full)" => run_mode(:full, seed) +}.map { |label, arrivals| [label, report(label, arrivals)] } + +puts "scoreboard (peak herd, smaller is safer):" +peaks.each { |label, peak| puts format(" %-32s %2d of %d", label, peak, WORKERS) } +puts +puts "full jitter trades punctuality for survival: retries land anywhere" +puts "in [0, delay], so some come back early and few come back TOGETHER." +puts "when the upstream is already hurting, together is the only wrong time." From a7cdd8cc684c8dd5a1bbd75d11d88e45b94fc069 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 7 Jul 2026 01:34:17 +0000 Subject: [PATCH 22/36] docs: refactor receipts example (Sandi Metz round 6) The god-join plan refactored in two small steps, each with a critic verdict and a measured receipt - including the honest one: step 1 removed the smell and cost 30ms before step 2 paid it back. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01FeZLdZ3M4q4cho4DZPfSkF --- docs/perspectives/round-6/09-sandimetz.md | 58 ++++++++++++++++ examples/refactor_receipts.rb | 82 +++++++++++++++++++++++ 2 files changed, 140 insertions(+) create mode 100644 docs/perspectives/round-6/09-sandimetz.md create mode 100644 examples/refactor_receipts.rb diff --git a/docs/perspectives/round-6/09-sandimetz.md b/docs/perspectives/round-6/09-sandimetz.md new file mode 100644 index 0000000..e178511 --- /dev/null +++ b/docs/perspectives/round-6/09-sandimetz.md @@ -0,0 +1,58 @@ +# Round 6 field notes — Sandi Metz collects the receipts + +*Built: `examples/refactor_receipts.rb` — the god-join plan improved in +two small steps, with a critic verdict and a measured receipt after +each one.* + +## What I built and why + +My round-4 critic could point at the god join and prescribe staged +joins; this round performs the surgery *with receipts*. Three states, +each critiqued and each executed: + +``` +before: wall 121ms | fan-in 5 | critic: god task (5 deps) +step 1: wall 151ms | fan-in 3 | critic: no complaints +step 2: wall 121ms | fan-in 3 | critic: no complaints +``` + +The receipt I'm proudest of is the one that embarrassed my first +draft's closing line. I wrote "same speed at every step" — the +measurement said otherwise: **step 1 removed the smell and cost +30ms**, because staging the joins added a level to the critical path. +Step 2 paid it back by letting the report read the stages directly. +I've taught for years that refactoring steps should be *safe*; the +receipts add the adult correction — safe is not free, and pretending +otherwise is how refactoring gets a reputation for making things +slower. Intermediate steps may cost. Receipts *price* them, so the +team can decide to pause at step 1 (shippable, smell-free, 30ms +slower) with open eyes instead of discovering the regression in +production. + +## The method, distilled + +1. Run the critic — it names the smell and one next step. +2. Take the step. Small. The kind you could revert in one commit. +3. Collect the receipt: structure numbers AND behavior numbers. +4. Decide — continue, stop, or revert — from evidence. + +That loop is exactly the refactoring kata I run with classes and +tests, transplanted onto graphs and wall clocks. The framework's +contribution is that steps 1 and 3 are each a dozen lines over +`graph` and `execute_plan` — cheap enough that nobody skips them, +which is the only cheapness that changes behavior. + +## A note on the shapes themselves + +Step 2's final shape has *no* intermediate `join` at all — the report +reads the staged pairs directly via fan-in. The refactoring didn't +just split the god task; it eventually *deleted* the middleman. That's +the pattern from object design: extract, then notice the extraction +made someone redundant. Graphs shed roles the same way classes do, +one safe step at a time. + +## Verdict + +The critic found it, the steps fixed it, the receipts priced it, and +my own summary got corrected by the data — a complete refactoring +story, including the humility. Ship the loop, not just the tools. diff --git a/examples/refactor_receipts.rb b/examples/refactor_receipts.rb new file mode 100644 index 0000000..dc74683 --- /dev/null +++ b/examples/refactor_receipts.rb @@ -0,0 +1,82 @@ +# frozen_string_literal: true + +# Refactor Receipts: the god-join plan from the graph critic, improved +# in two small steps - with a receipt after each one. Every step shows +# the smells found, the structure numbers, and the measured wall time, +# because "I made it better" is a claim and receipts are evidence. +# +# bundle exec ruby examples/refactor_receipts.rb +# +# Runs offline; each task is 30ms of simulated IO. + +require_relative "../lib/agentic" + +UNIT = 0.03 + +# Three versions of the same pipeline: five ingests feeding a report +SHAPES = { + "before: the god join" => { + "join" => %w[ingest_a ingest_b ingest_c ingest_d ingest_e], + "report" => %w[join] + }, + "step 1: stage the pairs" => { + "join_ab" => %w[ingest_a ingest_b], + "join_cde" => %w[ingest_c ingest_d ingest_e], + "join" => %w[join_ab join_cde], + "report" => %w[join] + }, + "step 2: report reads the stages" => { + "join_ab" => %w[ingest_a ingest_b], + "join_cde" => %w[ingest_c ingest_d ingest_e], + "report" => %w[join_ab join_cde] + } +}.freeze + +def build(deps) + orchestrator = Agentic::PlanOrchestrator.new(concurrency_limit: 4) + names = (%w[ingest_a ingest_b ingest_c ingest_d ingest_e] + deps.keys).uniq + tasks = names.to_h { |n| [n, Agentic::Task.new(description: n, agent_spec: {"name" => n, "instructions" => "work"})] } + names.each do |name| + orchestrator.add_task(tasks[name], (deps[name] || []).map { |d| tasks.fetch(d) }, + agent: ->(_t) { sleep(UNIT) || :ok }) + end + orchestrator +end + +def critique(graph) + dependencies = graph[:dependencies] + smells = [] + dependencies.each do |id, deps| + smells << "god task (#{deps.size} deps)" if deps.size >= 4 + end + depth = {} + measure = ->(id) { depth[id] ||= 1 + (dependencies[id].map { |d| measure.call(d) }.max || 0) } + max_depth = dependencies.keys.map { |id| measure.call(id) }.max + smells << "deep chain (#{max_depth} levels)" if max_depth >= 5 + [smells, max_depth, dependencies.values.map(&:size).max] +end + +puts "REFACTOR RECEIPTS (five ingests -> report, 30ms per task)" +puts + +SHAPES.each do |label, deps| + orchestrator = build(deps) + smells, depth, fan_in = critique(orchestrator.graph) + result = orchestrator.execute_plan + + puts " #{label}" + puts format(" wall %3dms | depth %d | max fan-in %d | tasks %d", + result.execution_time * 1000, depth, fan_in, orchestrator.graph[:tasks].size) + if smells.empty? + puts " critic: no complaints" + else + smells.each { |smell| puts " critic: #{smell}" } + end + puts +end + +puts "read the receipts honestly: step 1 removed the smell but COST 30ms" +puts "(the extra join level) - a receipt you'd never notice without the" +puts "measurement. step 2 pays it back by letting the report read the" +puts "stages directly. intermediate steps may cost; receipts price them," +puts "and every step was still a shippable state." From 516e15eb12e1786c83926ee241415ca01c979ff2 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 7 Jul 2026 01:35:44 +0000 Subject: [PATCH 23/36] docs: cost estimator example (ankane round 6) Prices the declared graph against a model pricing table before execution, gates on budget with a suggested fix, and reconciles estimates against seeded actuals - spend as a gate and feedback loop. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01FeZLdZ3M4q4cho4DZPfSkF --- docs/perspectives/round-6/10-ankane.md | 61 ++++++++++++++++++ examples/cost_estimator.rb | 88 ++++++++++++++++++++++++++ 2 files changed, 149 insertions(+) create mode 100644 docs/perspectives/round-6/10-ankane.md create mode 100644 examples/cost_estimator.rb diff --git a/docs/perspectives/round-6/10-ankane.md b/docs/perspectives/round-6/10-ankane.md new file mode 100644 index 0000000..d44ade7 --- /dev/null +++ b/docs/perspectives/round-6/10-ankane.md @@ -0,0 +1,61 @@ +# Round 6 field notes — Andrew Kane prices the plan + +*Built: `examples/cost_estimator.rb` — price the graph before running +it, gate on budget, reconcile estimate against actuals after.* + +## What I built and why + +Every LLM team I talk to discovers their spend from the invoice. The +estimator moves that discovery to *before the first token*: each task +carries `{model:, est_tokens:}` in its payload, the pre-flight walks +`graph[:tasks]` against a pricing table, and a budget gate refuses the +plan while refusing is still free: + +``` +TOTAL ~57.4c GATE: under budget, proceeding. +``` + +Run it with a 30c budget instead and the gate exits 1 *with the fix +suggested* ("downgrade 'draft responses' to the small model") — a +refusal that teaches beats a refusal that scolds. Then the +reconciliation, because estimates are hypotheses: + +``` +draft responses est 36.0c actual 49.8c (+38%) +TOTAL est 57.4c actual 71.7c +``` + +The demo's seeded drift made the point better than I planned: the run +passed the gate at 57c and *actually cost* 71c. That's not a bug in +the gate — that's the argument for the feedback loop printed in the +last line. Estimates without reconciliation decay into folklore; +actuals fed back into `est_tokens` make next month's gate honest. + +## Design notes + +- The pre-flight reads `graph[:tasks]` — pricing happens on the + *declared* plan, no execution machinery involved. Pairs naturally + with Xavier's wire format: price a plan.json in CI before a human + approves the deploy that runs it. +- Payload as the cost-metadata carrier is the pattern working as + designed: the framework never learns what `est_tokens` means, and + doesn't need to. Opaque payloads age well. +- Real-world wiring is small: `est_tokens` from a tokenizer count on + your prompts, actuals from the LLM response's usage block (the + `GenerationStats` class is already sitting in this gem waiting for + exactly this), pricing table from your provider's page. The shape + here is the product; the lambdas are placeholders. + +## What I'd ship next + +`agentic-budget`: the gate as a lifecycle hook (refuse at plan start, +running total during, alert at 80%), reconciliation into Perham's +journal so history accumulates, and a `--price plan.json` CLI mode. +Weekend-sized, invoice-shaped. + +## Verdict + +The plan can be priced before it runs and audited after — spend +became a gate and a feedback loop instead of a surprise. Six rounds +in, the gem's examples now cover the full lifecycle of an agent plan: +design, review, price, run, observe, resume, and bill. diff --git a/examples/cost_estimator.rb b/examples/cost_estimator.rb new file mode 100644 index 0000000..402629f --- /dev/null +++ b/examples/cost_estimator.rb @@ -0,0 +1,88 @@ +# frozen_string_literal: true + +# The Cost Estimator: price the plan BEFORE running it - per-task token +# estimates times a pricing table - gate on budget, then reconcile +# estimate against actuals afterward. LLM plans spend real money; +# nobody should learn the bill from the invoice. +# +# bundle exec ruby examples/cost_estimator.rb [budget_cents] +# +# Runs offline; actual token usage is seeded simulation. + +require_relative "../lib/agentic" + +# $/1M tokens, input+output blended for the demo +PRICING = { + "small" => 0.40, + "large" => 6.00 +}.freeze + +JOBS = { + "classify tickets" => {model: "small", est_tokens: 40_000}, + "summarize threads" => {model: "small", est_tokens: 120_000}, + "draft responses" => {model: "large", est_tokens: 60_000}, + "review drafts" => {model: "large", est_tokens: 25_000} +}.freeze + +def cents(model, tokens) + (PRICING.fetch(model) * tokens / 1_000_000 * 100) +end + +budget_cents = (ARGV.first || 60).to_i +rng = Random.new(20260707) + +orchestrator = Agentic::PlanOrchestrator.new(concurrency_limit: 2) +tasks = JOBS.map do |name, job| + task = Agentic::Task.new( + description: name, + agent_spec: {"name" => name, "instructions" => "spend wisely"}, + payload: job + ) + orchestrator.add_task(task, agent: ->(t) { + sleep(0.01) + # Actuals drift from estimates, as actuals do (+/- up to 40%) + drift = 0.8 + rng.rand * 0.6 + {tokens: (t.payload[:est_tokens] * drift).round} + }) + task +end + +# --- pre-flight: price the graph before any token is spent ------------------- +estimate = orchestrator.graph[:tasks].values.sum { |t| cents(t.payload[:model], t.payload[:est_tokens]) } + +puts "COST ESTIMATOR (budget: #{budget_cents}c)" +puts +puts " pre-flight estimate:" +JOBS.each do |name, job| + puts format(" %-20s %-6s ~%6d tokens ~%5.1fc", name, job[:model], job[:est_tokens], + cents(job[:model], job[:est_tokens])) +end +puts format(" %-20s %28s %5.1fc", "TOTAL", "", estimate) +puts + +if estimate > budget_cents + puts " GATE: estimate exceeds budget - plan refused before spending a cent." + puts " (raise the budget or downgrade 'draft responses' to the small model)" + exit 1 +end +puts " GATE: under budget, proceeding." +puts + +# --- the run, then the reconciliation ---------------------------------------- +result = orchestrator.execute_plan + +puts " reconciliation (estimate vs actual):" +total_actual = 0.0 +tasks.each do |task| + job = task.payload + actual_tokens = result.results[task.id].output[:tokens] + actual = cents(job[:model], actual_tokens) + total_actual += actual + est = cents(job[:model], job[:est_tokens]) + drift_pct = 100.0 * (actual - est) / est + puts format(" %-20s est %5.1fc actual %5.1fc (%+.0f%%)", task.description, est, actual, drift_pct) +end +puts format(" %-20s est %5.1fc actual %5.1fc", "TOTAL", estimate, total_actual) +puts +puts " feed actuals back into est_tokens and next month's estimates" +puts " stop being folklore. budgets want feedback loops, not faith." From 49de058a68405de33b1080f46adac02046296664 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 7 Jul 2026 01:37:23 +0000 Subject: [PATCH 24/36] docs: round-6 index and findings; regenerate examples catalog Adds the round-6 table to the perspectives README: plans as artifacts (narratable, serializable, priceable, diffable), declarations as testable claims, and the next asks. Examples catalog regenerated at 50 entries. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01FeZLdZ3M4q4cho4DZPfSkF --- docs/perspectives/README.md | 42 +++++++++++++++++++++++++++++++++++++ examples/README.md | 10 +++++++++ 2 files changed, 52 insertions(+) diff --git a/docs/perspectives/README.md b/docs/perspectives/README.md index 840653d..68099b9 100644 --- a/docs/perspectives/README.md +++ b/docs/perspectives/README.md @@ -106,6 +106,48 @@ more experiments followed: | 9 | Sandi Metz | Three shapes — chain vs star vs staged, chosen by evidence | `examples/three_shapes.rb` | [round-5/09-sandimetz.md](round-5/09-sandimetz.md) | | 10 | Andrew Kane | Examples index — self-maintaining signage for a 40-example gallery | `examples/examples_index.rb` | [round-5/10-ankane.md](round-5/10-ankane.md) | +## Round 6 — plans as artifacts + +The round-5 asks shipped as a release (`graph[:order]` via Kahn's +algorithm, labeled `graph[:edges]`, structured rules with +`fields:`/`rule_violations`, `backoff_jitter: :full`, and windowed +rate limits `RateLimit.new(30, per: 60)`), four examples were +modernized onto them, and ten more experiments followed: + +| # | Persona | Built on the round-6 release | Run it | Field notes | +|---|---------|------------------------------|--------|-------------| +| 1 | Matz | Plan tour — the plan narrated as prose, before it runs | `examples/plan_tour.rb` | [round-6/01-matz.md](round-6/01-matz.md) | +| 2 | DHH | Deploy train — the unhappy path as the product | `examples/deploy_train.rb` | [round-6/02-dhh.md](round-6/02-dhh.md) | +| 3 | Aaron Patterson | Perf diff — did the PR make it worse, path-qualified | `examples/perf_diff.rb` | [round-6/03-tenderlove.md](round-6/03-tenderlove.md) | +| 4 | Xavier Noria | Plan round-trip — graph → JSON → graph with isomorphism proof | `examples/plan_roundtrip.rb` | [round-6/04-fxn.md](round-6/04-fxn.md) | +| 5 | Samuel Williams | Quota keeper — ceiling physics vs window physics, 61ms vs 601ms | `examples/quota_keeper.rb` | [round-6/05-ioquatix.md](round-6/05-ioquatix.md) | +| 6 | Jeremy Evans | Rule prober — field declarations audited; a lying rule caught | `examples/rule_prober.rb` | [round-6/06-jeremyevans.md](round-6/06-jeremyevans.md) | +| 7 | Piotr Solnica | API reference — docs from the contracts that validate the calls | `examples/api_reference.rb` | [round-6/07-solnic.md](round-6/07-solnic.md) | +| 8 | Mike Perham | Jitter shootout — none/equal/full on one scoreboard: 40/19/13 | `examples/jitter_shootout.rb` | [round-6/08-mperham.md](round-6/08-mperham.md) | +| 9 | Sandi Metz | Refactor receipts — the god join dissolved in priced steps | `examples/refactor_receipts.rb` | [round-6/09-sandimetz.md](round-6/09-sandimetz.md) | +| 10 | Andrew Kane | Cost estimator — the plan priced before it runs, reconciled after | `examples/cost_estimator.rb` | [round-6/10-ankane.md](round-6/10-ankane.md) | + +### What round 6 surfaced + +1. **Plans became artifacts**: narratable (tour), serializable with an + isomorphism proof (round-trip), priceable before execution (cost + gate), and diffable across runs (perf diff). The graph accessor's + second round turned topology into a first-class document. +2. **Declarations became testable claims**: rule `fields:` shipped as + UI plumbing and immediately became an auditable specification — the + prober caught a seeded lying rule that would have misdirected form + highlighting. +3. **One contract, five behaviors**: validate, reject, explain, + document, audit — the same declaration now feeds all five. +4. **Honest prose corrections**: two personas (DHH, Sandi) had their + example copy corrected by their own measurements — the tools are + now good enough to disagree with their authors. +5. **Next asks**: composing a windowed and a concurrency limiter as + one object, journal-fed baselines for the perf diff, OpenAPI + emission from contracts, custom RNG injection for retry policies, + and a `graph`-level depth/fan-in stats helper (Sandi's third + strike). + ### What round 5 surfaced 1. **The graph accessor compounded immediately**: one round old, it fed diff --git a/examples/README.md b/examples/README.md index 597b1a0..c02c1aa 100644 --- a/examples/README.md +++ b/examples/README.md @@ -6,13 +6,16 @@ not this file. | Example | What it shows | |---------|---------------| +| `api_reference.rb` | The API Reference Generator: walk the registry, emit reference docs for every capability - types, enums, bounds, policie... | | `burst_absorber.rb` | The Burst Absorber: three waves of requests slam a credential with a ceiling of 3 (Agentic::RateLimit - this round's rel... | | `changelog_scout.rb` | The Changelog Scout: reads real git history, classifies every commit through a contract-checked capability, and drafts t... | | `collaboration_tracer.rb` | The Collaboration Tracer: lifecycle hooks record every message the orchestrator sends and every reply that comes back, t... | | `command_bus.rb` | The Command Bus: every command is a composed capability with its OWN declared contract (new in this round - compositions... | | `contract_fuzzer.rb` | The Contract Fuzzer: for every registered capability, generate inputs that SHOULD pass its declared contract and mutatio... | +| `cost_estimator.rb` | The Cost Estimator: price the plan BEFORE running it - per-task token estimates times a pricing table - gate on budget, ... | | `coupling_cartographer.rb` | The Coupling Cartographer: which files lean on which? Every file is surveyed for the constants it DEFINES and the consta... | | `critical_path.rb` | The Critical Path: after a run, combine the graph topology with measured durations to find the chain of tasks that deter... | +| `deploy_train.rb` | The Deploy Train: lint -> test -> build -> canary -> ship, where a red gate stops the train and everything behind it rep... | | `doc_coverage.rb` | The Documentation Surveyor: measures YARD comment coverage for every public method in a lib/ tree. One survey task per f... | | `dungeon_crawl.rb` | The Dungeon Crawl: a quest is a plan, rooms are tasks, and doors are dependencies. The map is drawn from the orchestrato... | | `durable_batch.rb` | The Durable Batch: six billable "LLM calls" run under an ExecutionJournal. Mid-batch, the process dies for real - exit!,... | @@ -25,17 +28,24 @@ not this file. | `graph_critic.rb` | The Graph Critic: reviews a plan's dependency structure BEFORE it runs, the way you'd review a class diagram. God tasks,... | | `haiku_agent.rb` | The three-line agent. Run me with no API key at all: | | `invariant_sentinel.rb` | The Invariant Sentinel: domain invariants checked after EVERY task, from a lifecycle hook. When a task leaves the world ... | +| `jitter_shootout.rb` | The Jitter Shootout: none vs equal (+/-25%, the default) vs full (uniform over [0, delay], new this round) - same forty ... | | `kanban_board.rb` | The Kanban Board: a plan rendered as the three columns everyone actually understands - To Do, Doing, Done - reprinted at... | | `knee_finder.rb` | The Knee Finder: runs the same plan at increasing concurrency limits, measures wall time and total queue-wait via the ta... | | `latency_lab.rb` | The Latency Lab: 20 simulated LLM calls (200ms of IO each) executed through the orchestrator at different concurrency li... | | `live_dashboard.rb` | The Live Dashboard: lifecycle hooks publish events onto an Async::Queue; a consumer task IN THE SAME REACTOR renders the... | | `namespace_cartographer.rb` | The Namespace Cartographer: maps a gem's constant tree and audits every file against the constant Zeitwerk expects it to... | +| `perf_diff.rb` | The Perf Diff: run the plan before and after a change, diff per-task durations, and flag regressions - with the one qual... | | `performance_detective.rb` | The Performance Detective: one task per Ruby file in lib/, fanned out through the orchestrator, each dissecting a file f... | | `plan_diagram.rb` | The Plan Diagrammer: any orchestrator's graph, emitted as Mermaid - paste it into a README, GitHub renders it, and the d... | | `plan_gantt.rb` | The Plan Gantt: lifecycle hooks timestamp every task, then the run is rendered as an ASCII timeline - where your wall cl... | +| `plan_roundtrip.rb` | The Round Trip: serialize a plan's graph to JSON, rebuild a fresh orchestrator from the JSON, and prove the rebuilt topo... | +| `plan_tour.rb` | The Plan Tour: hand any orchestrator to the guide and it narrates the plan as prose - first this, then that, meanwhile t... | +| `quota_keeper.rb` | The Quota Keeper: the same 20 requests through two different laws. A concurrency ceiling ("3 in flight") models connecti... | | `readme_verifier.rb` | The README Verifier: every ruby code fence in the README is a promise. This extracts them all, syntax-checks each with P... | +| `refactor_receipts.rb` | Refactor Receipts: the god-join plan from the graph critic, improved in two small steps - with a receipt after each one.... | | `refactoring_dojo.rb` | The Refactoring Dojo: a student submits a method, three critic agents review it from three distinct perspectives, and th... | | `renga_circle.rb` | A renga circle: three poet agents compose a linked-verse poem, each verse responding to the one before it. The dependenc... | +| `rule_prober.rb` | The Rule Prober: structured rules declare which fields they read - so now that claim can be AUDITED. For each rule, pert... | | `schema_advisor.rb` | The Schema Advisor: give it a schema and a query log, get back the advisories a careful DBA would write - each rule its ... | | `setup_doctor.rb` | The Setup Doctor: every onboarding wiki page is a bug. This runs the checks a README asks a new hire to do by hand - rub... | | `shared_rate_limit.rb` | The Shared Rate Limit: two plans run concurrently in one reactor, but the API key they share allows only 3 requests in f... | From 622a0cc63100745862d14b8e819ee7a836f5b97b Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 7 Jul 2026 11:15:44 +0000 Subject: [PATCH 25/36] feat: deliver the round-6 asks The prioritized asks from the round-6 field notes, implemented: - RateLimit#and composes limiters: a windowed quota and a concurrency ceiling enforced together (quota.and(pool)), nesting flattened - graph[:stats]: per-task depth, max depth, and max fan-in computed once in the snapshot - three examples had hand-rolled this walk - ExecutionJournal replay exposes per-task durations keyed by description, giving performance tooling a baseline source for free - CapabilitySpecification#to_json_schema emits either contract side as draft-07 JSON Schema (types, enums, bounds, non-empty, descriptions) - Retry policies accept rng: (any object with #rand) so jitter timing is injectable and reproducible in tests refactor_receipts and graph_critic modernized onto graph[:stats]. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01FeZLdZ3M4q4cho4DZPfSkF --- README.md | 19 ++-- examples/graph_critic.rb | 5 +- examples/refactor_receipts.rb | 13 ++- lib/agentic/capability_specification.rb | 43 ++++++++ lib/agentic/execution_journal.rb | 15 ++- lib/agentic/plan_orchestrator.rb | 28 ++++- lib/agentic/rate_limit.rb | 41 +++++++ spec/agentic/round7_features_spec.rb | 136 ++++++++++++++++++++++++ 8 files changed, 277 insertions(+), 23 deletions(-) create mode 100644 spec/agentic/round7_features_spec.rb diff --git a/README.md b/README.md index 2a73b25..2e2af84 100644 --- a/README.md +++ b/README.md @@ -248,15 +248,22 @@ default so fleets don't retry in lockstep (`backoff_jitter: :full` draws uniformly from `[0, delay]` for the hardest herd-flattening). `PlanOrchestrator#graph` returns a frozen snapshot of the plan's topology -— including `graph[:order]` (topological sort) and `graph[:edges]` -(labeled by `needs:` names) — for tools that render, review, or analyze -plans. `Agentic::RateLimit` is a credential-scoped ceiling shareable -across plans and clients (`Agentic::LlmClient.new(config, limiter: rate_limit)`): -concurrent by default, or a rolling-window quota with -`RateLimit.new(30, per: 60)`. Cross-field rules may be structured — +— including `graph[:order]` (topological sort), `graph[:edges]` (labeled +by `needs:` names), and `graph[:stats]` (per-task depth, max depth, max +fan-in) — for tools that render, review, or analyze plans. +`Agentic::RateLimit` is a credential-scoped ceiling shareable across +plans and clients (`Agentic::LlmClient.new(config, limiter: rate_limit)`): +concurrent by default, a rolling-window quota with +`RateLimit.new(30, per: 60)`, or both laws at once via composition — +`quota.and(pool)`. Cross-field rules may be structured — `rules: {air_weight_limit: {message: "...", fields: [:mode, :weight], check: ->(i) {...}}}` — and `ValidationError#rule_violations` reports each broken rule with its identifier, message, and the fields it reads. +`CapabilitySpecification#to_json_schema` emits a contract side as +draft-07 JSON Schema for OpenAPI and validator toolchains; journal +replays expose per-task `durations` keyed by description (performance +baselines for free); and retry policies accept an injected `rng:` for +reproducible jitter timing. ### The concurrency contract diff --git a/examples/graph_critic.rb b/examples/graph_critic.rb index 31161c8..77cf577 100644 --- a/examples/graph_critic.rb +++ b/examples/graph_critic.rb @@ -49,9 +49,8 @@ def task_named(name) dependents = Hash.new { |h, k| h[k] = [] } dependencies.each { |task_id, deps| deps.each { |dep| dependents[dep] << task_id } } -depth_of = lambda do |task_id, memo = {}| - memo[task_id] ||= 1 + (dependencies[task_id].map { |dep| depth_of.call(dep, memo) }.max || 0) -end +# Depth now ships precomputed in the snapshot +depth_of = ->(task_id) { graph[:stats][:depth][task_id] } findings = [] diff --git a/examples/refactor_receipts.rb b/examples/refactor_receipts.rb index dc74683..4ccb22e 100644 --- a/examples/refactor_receipts.rb +++ b/examples/refactor_receipts.rb @@ -43,17 +43,16 @@ def build(deps) orchestrator end +# The depth/fan-in walk this example used to hand-roll now ships as +# graph[:stats] - the critique is just thresholds over facts def critique(graph) - dependencies = graph[:dependencies] + stats = graph[:stats] smells = [] - dependencies.each do |id, deps| + graph[:dependencies].each do |_id, deps| smells << "god task (#{deps.size} deps)" if deps.size >= 4 end - depth = {} - measure = ->(id) { depth[id] ||= 1 + (dependencies[id].map { |d| measure.call(d) }.max || 0) } - max_depth = dependencies.keys.map { |id| measure.call(id) }.max - smells << "deep chain (#{max_depth} levels)" if max_depth >= 5 - [smells, max_depth, dependencies.values.map(&:size).max] + smells << "deep chain (#{stats[:max_depth]} levels)" if stats[:max_depth] >= 5 + [smells, stats[:max_depth], stats[:max_fan_in]] end puts "REFACTOR RECEIPTS (five ingests -> report, 30ms per task)" diff --git a/lib/agentic/capability_specification.rb b/lib/agentic/capability_specification.rb index bf6e356..5cfc934 100644 --- a/lib/agentic/capability_specification.rb +++ b/lib/agentic/capability_specification.rb @@ -89,6 +89,49 @@ def self.from_h(hash) ) end + # Emits the declared contract as a JSON Schema object, so capability + # contracts plug into OpenAPI tooling, JSON validators, and every + # schema-aware toolchain without a bespoke exporter + # @param side [Symbol] :inputs or :outputs + # @return [Hash] A JSON Schema (draft-07 compatible) as a plain hash + def to_json_schema(side = :inputs) + declared = (side == :outputs) ? outputs : inputs + + properties = declared.to_h do |name, decl| + schema = {} + schema["type"] = JSON_SCHEMA_TYPES.fetch(decl[:type], nil) if decl[:type] + schema.delete("type") if schema["type"].nil? + schema["description"] = decl[:description] if decl[:description] + schema["enum"] = decl[:enum] if decl[:enum] + schema["minimum"] = decl[:min] if decl[:min] + schema["maximum"] = decl[:max] if decl[:max] + if decl[:non_empty] + schema[(decl[:type] == "array") ? "minItems" : "minLength"] = 1 + end + [name.to_s, schema] + end + + { + "$schema" => "http://json-schema.org/draft-07/schema#", + "title" => "#{name} #{side}", + "type" => "object", + "required" => declared.select { |_, decl| decl[:required] }.keys.map(&:to_s), + "properties" => properties, + "additionalProperties" => true + } + end + + # Contract type names to JSON Schema type names + JSON_SCHEMA_TYPES = { + "string" => "string", + "number" => "number", + "integer" => "integer", + "boolean" => "boolean", + "array" => "array", + "object" => "object", + "hash" => "object" + }.freeze + # Get the capability requirements as a human-readable string # @return [String] The capability requirements def requirements_description diff --git a/lib/agentic/execution_journal.rb b/lib/agentic/execution_journal.rb index 103044a..9770484 100644 --- a/lib/agentic/execution_journal.rb +++ b/lib/agentic/execution_journal.rb @@ -23,9 +23,16 @@ module Agentic class ExecutionJournal # Replayed journal state: everything a resuming process needs to know ReplayedState = Struct.new( - :plan_id, :status, :completed_task_ids, :failed_task_ids, :outputs, :failures, :events, :descriptions, + :plan_id, :status, :completed_task_ids, :failed_task_ids, :outputs, :failures, :events, :descriptions, :durations, keyword_init: true ) do + # Task durations keyed by description - the natural baseline source + # for performance regression tooling + # @return [Hash{String=>Float}] Description => seconds (latest wins) + def durations_by_description + durations + end + # @param key [String] A task id or a task description (descriptions # act as idempotency keys across runs, since ids are per-run) # @return [Boolean] True if the journal records a success for the task @@ -100,7 +107,8 @@ def self.replay(path:) outputs: {}, failures: {}, events: [], - descriptions: {} + descriptions: {}, + durations: {} ) return state unless File.exist?(path) @@ -122,6 +130,9 @@ def self.replay(path:) state.failed_task_ids.delete(task_id) state.failures.delete(task_id) state.outputs[task_id] = entry[:output] + if entry[:description] && entry[:duration] + state.durations[entry[:description]] = entry[:duration] + end when "task_failed" task_id = entry[:task_id] unless state.completed_task_ids.include?(task_id) diff --git a/lib/agentic/plan_orchestrator.rb b/lib/agentic/plan_orchestrator.rb index 924ee6d..fc0604e 100644 --- a/lib/agentic/plan_orchestrator.rb +++ b/lib/agentic/plan_orchestrator.rb @@ -107,12 +107,27 @@ def graph deps.map { |dep_id| {from: dep_id, to: task_id, label: labels[[dep_id, task_id]]}.freeze } }.freeze + order = topological_order + + # Structural stats, computed once so every graph tool stops + # hand-rolling the same walk: per-task depth (1 = root), the + # longest chain, and the widest fan-in + depth = {} + order.each do |task_id| + depth[task_id] = 1 + (@dependencies[task_id].map { |dep| depth[dep] || 0 }.max || 0) + end + { tasks: @tasks.dup.freeze, dependencies: @dependencies.transform_values { |deps| deps.dup.freeze }.freeze, needs: @task_needs.transform_values { |named| named.dup.freeze }.freeze, - order: topological_order.freeze, - edges: edges + order: order.freeze, + edges: edges, + stats: { + depth: depth.freeze, + max_depth: depth.values.max || 0, + max_fan_in: @dependencies.values.map(&:size).max || 0 + }.freeze }.freeze end @@ -269,13 +284,16 @@ def apply_retry_backoff(task:) # Apply jitter (on by default) so fleets don't retry in lockstep. # true = equal jitter (+/-25%); :full = full jitter (uniform over - # [0, delay]), which flattens synchronized herds much harder + # [0, delay]), which flattens synchronized herds much harder. + # An injected rng: (any object with #rand) makes timing testable. + rng = @retry_policy[:rng] case @retry_policy[:backoff_jitter] when :full - delay = rand(0.0..delay) + delay = rng ? rng.rand(0.0..delay) : rand(0.0..delay) when true jitter_factor = 0.25 # Default 25% jitter - jitter = rand(-delay * jitter_factor..delay * jitter_factor) + band = -delay * jitter_factor..delay * jitter_factor + jitter = rng ? rng.rand(band) : rand(band) delay = [delay + jitter, 0].max end diff --git a/lib/agentic/rate_limit.rb b/lib/agentic/rate_limit.rb index 105766c..28aa9b2 100644 --- a/lib/agentic/rate_limit.rb +++ b/lib/agentic/rate_limit.rb @@ -54,6 +54,47 @@ def acquire # @return [Integer] Acquisitions currently inside the ceiling attr_reader :in_flight + # Composes this limit with another: the block runs only inside BOTH. + # Production APIs usually enforce two laws at once - a billed quota + # (window) and a connection limit (ceiling): + # + # quota = Agentic::RateLimit.new(30, per: 60) + # pool = Agentic::RateLimit.new(3) + # limit = quota.and(pool) # acquires quota first, then a connection + # + # @param other [RateLimit, Composite] The inner limit + # @return [Composite] A limiter enforcing both + def and(other) + Composite.new(self, other) + end + + # Two or more limits acquired in order, released in reverse + class Composite + # @return [Array] The composed limits, outermost first + attr_reader :limits + + # @param limits [Array] Outermost first + def initialize(*limits) + @limits = limits.flat_map { |l| l.is_a?(Composite) ? l.limits : [l] } + end + + # Runs the block inside every composed limit + # @yield The rate-limited work + # @return [Object] The block's return value + def acquire(&block) + @limits.reverse.reduce(block) { |inner, limit| + -> { limit.acquire(&inner) } + }.call + end + + # Composes further: quota.and(pool).and(burst) + # @param other [RateLimit, Composite] The next inner limit + # @return [Composite] + def and(other) + Composite.new(self, other) + end + end + private def track diff --git a/spec/agentic/round7_features_spec.rb b/spec/agentic/round7_features_spec.rb new file mode 100644 index 0000000..88649e1 --- /dev/null +++ b/spec/agentic/round7_features_spec.rb @@ -0,0 +1,136 @@ +# frozen_string_literal: true + +require "spec_helper" +require "tmpdir" + +RSpec.describe "round 7 framework features" do + def task_named(description) + Agentic::Task.new( + description: description, + agent_spec: {"name" => "worker", "instructions" => "work"} + ) + end + + describe "RateLimit composition" do + it "enforces both a window and a ceiling at once" do + quota = Agentic::RateLimit.new(4, per: 0.1) + pool = Agentic::RateLimit.new(2) + limit = quota.and(pool) + + Sync do + 8.times.map { Async { limit.acquire { sleep(0.01) } } }.each(&:wait) + end + + expect(pool.high_water).to eq(2) # the ceiling held + expect(quota.high_water).to be <= 4 + end + + it "flattens nested composition" do + a = Agentic::RateLimit.new(1) + b = Agentic::RateLimit.new(2) + c = Agentic::RateLimit.new(3) + + expect(a.and(b).and(c).limits).to eq([a, b, c]) + end + end + + describe "graph[:stats]" do + it "reports per-task depth, max depth, and max fan-in" do + orchestrator = Agentic::PlanOrchestrator.new + roots = 3.times.map { |i| task_named("root-#{i}") } + join = task_named("join") + tail = task_named("tail") + + roots.each { |t| orchestrator.add_task(t, agent: ->(_t) { :ok }) } + orchestrator.add_task(join, roots, agent: ->(_t) { :ok }) + orchestrator.add_task(tail, [join], agent: ->(_t) { :ok }) + + stats = orchestrator.graph[:stats] + + expect(stats[:depth][roots.first.id]).to eq(1) + expect(stats[:depth][join.id]).to eq(2) + expect(stats[:depth][tail.id]).to eq(3) + expect(stats[:max_depth]).to eq(3) + expect(stats[:max_fan_in]).to eq(3) + end + end + + describe "journal durations" do + it "replays task durations keyed by description" do + Dir.mktmpdir do |dir| + path = File.join(dir, "durations.jsonl") + journal = Agentic::ExecutionJournal.new(path: path) + + orchestrator = Agentic::PlanOrchestrator.new(lifecycle_hooks: journal.lifecycle_hooks) + task = task_named("timed-work") + orchestrator.add_task(task, agent: ->(_t) { sleep(0.02) || :ok }) + orchestrator.execute_plan + + state = Agentic::ExecutionJournal.replay(path: path) + + expect(state.durations).to have_key("timed-work") + expect(state.durations["timed-work"]).to be >= 0.015 + end + end + end + + describe "retry policy rng injection" do + it "draws jitter from the injected rng, making timing reproducible" do + draws = [] + fake_rng = Class.new { + def initialize(draws) = @draws = draws + + def rand(range) + @draws << range + (range.begin + range.end) / 2.0 + end + }.new(draws) + + orchestrator = Agentic::PlanOrchestrator.new( + retry_policy: {backoff_strategy: :constant, backoff_constant: 1.0, backoff_jitter: :full, rng: fake_rng} + ) + allow(orchestrator).to receive(:sleep) + + task = task_named("flaky") + task.retry_count = 1 + orchestrator.apply_retry_backoff(task: task) + + expect(draws).to eq([0.0..1.0]) + expect(orchestrator).to have_received(:sleep).with(0.5) + end + end + + describe "CapabilitySpecification#to_json_schema" do + let(:specification) do + Agentic::CapabilitySpecification.new( + name: "ship", description: "ships", version: "1.0.0", + inputs: { + speed: {type: "string", required: true, enum: %w[standard express], description: "Shipping tier"}, + quantity: {type: "number", required: true, min: 1, max: 100}, + items: {type: "array", non_empty: true}, + note: {type: "string"} + }, + outputs: {tracking: {type: "string", required: true}} + ) + end + + it "emits draft-07 JSON Schema for inputs" do + schema = specification.to_json_schema + + expect(schema["required"]).to contain_exactly("speed", "quantity") + expect(schema["properties"]["speed"]).to eq( + "type" => "string", "description" => "Shipping tier", "enum" => %w[standard express] + ) + expect(schema["properties"]["quantity"]).to include("minimum" => 1, "maximum" => 100) + expect(schema["properties"]["items"]).to include("minItems" => 1) + expect(schema["properties"]["note"]).to eq("type" => "string") + end + + it "emits the outputs side on request" do + schema = specification.to_json_schema(:outputs) + + expect(schema["title"]).to eq("ship outputs") + expect(schema["required"]).to eq(["tracking"]) + end + end +end From 1fc6613dae16ebd165f2256c7805ef53c089e587 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 7 Jul 2026 11:17:06 +0000 Subject: [PATCH 26/36] docs: plan fortune teller example (Matz round 7) graph[:stats] read as a palm - four checkable structural diagnoses in a mystic's robe. Field notes argue charm is a rigorous API test and ask for stats[:roots]/stats[:leaves]. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01FeZLdZ3M4q4cho4DZPfSkF --- docs/perspectives/round-7/01-matz.md | 56 +++++++++++++++++++ examples/plan_fortune.rb | 82 ++++++++++++++++++++++++++++ 2 files changed, 138 insertions(+) create mode 100644 docs/perspectives/round-7/01-matz.md create mode 100644 examples/plan_fortune.rb diff --git a/docs/perspectives/round-7/01-matz.md b/docs/perspectives/round-7/01-matz.md new file mode 100644 index 0000000..870a2f7 --- /dev/null +++ b/docs/perspectives/round-7/01-matz.md @@ -0,0 +1,56 @@ +# Round 7 field notes — Matz tells the plan's fortune + +*Built: `examples/plan_fortune.rb` — graph[:stats] read as a palm; +every fortune is a structural fact in a mystic's robe.* + +## What I built and why + +`graph[:stats]` shipped this round (my colleagues' third-strike ask), +and my first instinct with any new introspection API is to see whether +it can be *charming* — because charm is a rigorous test. A boring API +can hide behind utility; an API asked to entertain must be complete +enough to characterize its subject. The teller reads four lines of the +palm: + +``` +* You begin in many places at once (4 roots). +* Beware: all rivers flow through one gate (fan-in 4). +* I see a long road, 5 stations deep, in a caravan of only 8 - + the critical path knows your name. +* All ends in a single scroll (1 leaf). Tidy. +``` + +Every sentence is checkable: roots counted from empty dependency +lists, the gate from `stats[:max_fan_in]`, the long road from +`stats[:max_depth]` against task count, the scroll from edge +out-degrees. The robe is silk but the palmistry is arithmetic. And +notice what the fortune *is*, once you take the robe off: it's Sandi's +graph critic, DHH's honesty about bottlenecks, and Aaron's critical +path warning — the same diagnoses, delivered so the seeker smiles +while receiving them. Tone is an underrated API feature. People fix +what they enjoyed hearing about. + +## What the stats API got right + +- Depth arrives *per task* plus aggregated — the teller only needed + the aggregates, the perf tools need the per-task map, one snapshot + serves both. Providing the raw map alongside the summary is the + courteous shape for any stats API. +- The depth/tasks ratio ("more than half your journey walks single + file") required no new API — good primitives compose into new + observations freely. When a stats object makes you invent *derived* + metrics on the spot, it exposed the right primitives. + +## A small wish, as tradition demands + +Leaves (tasks nothing depends on) I computed by scanning edges; +roots by scanning for empty dependencies. Both are one-liners, but +they're the fortune-teller's bread and butter — `stats[:roots]` and +`stats[:leaves]` would round out the palm. Small ask, round 8. + +## Verdict + +Seven rounds in, the same graph can be executed, drawn, narrated, +priced, diffed — and now teased. A framework you can joke *with* +(not merely about) has crossed some threshold of maturity that +benchmarks don't measure. The ancestors approve. diff --git a/examples/plan_fortune.rb b/examples/plan_fortune.rb new file mode 100644 index 0000000..04ca3ad --- /dev/null +++ b/examples/plan_fortune.rb @@ -0,0 +1,82 @@ +# frozen_string_literal: true + +# The Plan Fortune Teller: reads your graph's palm - depth, fan-in, +# roots, breadth - and tells its fortune. Every fortune is a real +# structural fact wearing a mystic's robe; the entertainment is a +# delivery mechanism for the diagnosis. +# +# bundle exec ruby examples/plan_fortune.rb +# +# Runs offline. The stars are graph[:stats]. + +require_relative "../lib/agentic" + +def step(name) + Agentic::Task.new(description: name, agent_spec: {"name" => name, "instructions" => "toil"}) +end + +# A seeker arrives with their plan +orchestrator = Agentic::PlanOrchestrator.new +gather = 4.times.map { |i| step("gather-#{i}") } +sift = step("sift") +weigh = step("weigh") +judge = step("judge") +scribe = step("scribe") + +gather.each { |t| orchestrator.add_task(t) } +orchestrator.add_task(sift, gather) +orchestrator.add_task(weigh, [sift]) +orchestrator.add_task(judge, [weigh]) +orchestrator.add_task(scribe, [judge]) + +# --- the reading -------------------------------------------------------------- +graph = orchestrator.graph +stats = graph[:stats] +roots = graph[:dependencies].count { |_, deps| deps.empty? } +leaves = graph[:dependencies].keys.count { |id| graph[:edges].none? { |e| e[:from] == id } } +tasks = graph[:tasks].size + +fortunes = [] + +fortunes << if roots >= 4 + "You begin in many places at once, child of parallelism - " \ + "your mornings are wide (#{roots} roots)." +else + "You begin cautiously (#{roots} root#{(roots == 1) ? "" : "s"}) - " \ + "the fates smile on those who fan out." +end + +fortunes << if stats[:max_fan_in] >= 4 + "Beware: all rivers flow through one gate (fan-in #{stats[:max_fan_in]}). " \ + "When that gate falters, all waters still." +else + "Your joinings are modest (fan-in #{stats[:max_fan_in]}) - no single " \ + "gate holds your fate." +end + +fortunes << if stats[:max_depth] > tasks / 2 + "I see a long road, #{stats[:max_depth]} stations deep, in a caravan of " \ + "only #{tasks}. More than half your journey walks single file - " \ + "latency stalks you, and the critical path knows your name." +else + "Your road is short for your numbers (depth #{stats[:max_depth]} of " \ + "#{tasks}) - swiftness favors you." +end + +fortunes << if leaves == 1 + "All ends in a single scroll (1 leaf). Tidy. The ancestors approve of " \ + "plans that know what they produce." +else + "Your plan ends in #{leaves} places - be sure someone reads them all." +end + +puts "THE PLAN FORTUNE TELLER" +puts +puts " the seeker presents a plan of #{tasks} tasks..." +puts " the teller studies graph[:stats] (the palm never lies):" +puts +fortunes.each { |fortune| puts " * #{fortune}" } +puts +puts " cross my palm with a refactor and return: the fortune about the" +puts " long road is a real diagnosis - see refactor_receipts.rb for" +puts " the cure, and three_shapes.rb to choose your next incarnation." From 181231b40561127d5b4c775f8d49207f279d28f2 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 7 Jul 2026 11:18:14 +0000 Subject: [PATCH 27/36] docs: weekly check-in example (DHH round 7) Three days of journaled plans replayed into a Friday check-in: what shipped, what got stuck (with cross-run recovery detection via stable descriptions), and where the time went via the new durations. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01FeZLdZ3M4q4cho4DZPfSkF --- docs/perspectives/round-7/02-dhh.md | 57 ++++++++++++++++++++ examples/weekly_checkin.rb | 84 +++++++++++++++++++++++++++++ 2 files changed, 141 insertions(+) create mode 100644 docs/perspectives/round-7/02-dhh.md create mode 100644 examples/weekly_checkin.rb diff --git a/docs/perspectives/round-7/02-dhh.md b/docs/perspectives/round-7/02-dhh.md new file mode 100644 index 0000000..10c7ec6 --- /dev/null +++ b/docs/perspectives/round-7/02-dhh.md @@ -0,0 +1,57 @@ +# Round 7 field notes — DHH cancels the check-in meeting + +*Built: `examples/weekly_checkin.rb` — three days of journaled plans, +then the Friday check-in written from the replay instead of from +anyone's memory.* + +## What I built and why + +Basecamp's automatic check-ins replaced status meetings with a +question the tool asks so a human doesn't have to. This goes one +further: for work that runs as plans, **the journal already knows the +answers**. Three days of runs, one replay, and the check-in writes +itself: + +``` +What did you work on this week? <- completed_descriptions +Anything get stuck? <- task_failed events, with recovery + detected across runs +Where did the time go? <- durations (new this round) +``` + +The line I care about most: *"wed: email statements (smtp relay +refused) — recovered later in the week."* The journal noticed that +Wednesday's failure was retried and cleared on Friday, because +descriptions are stable across runs. That's the difference between a +log and a *narrative* — logs record events, narratives connect them, +and the connection is what a manager actually wants from a check-in. +Human status reports systematically forget Wednesday's failure by +Friday (self-report bias is undefeated); the journal doesn't. + +## The durations dividend + +`state.durations` shipped this round for Aaron's perf baselines, and +the check-in got its third question from it for free: "403ms of +tracked work; slowest was backfill invoices." One feature, requested +for regression testing, immediately answers a management question. +That's the recurring economics of this whole experiment: **data +captured for one consumer is data captured for all of them.** The +journal started as crash recovery (round 2), became resume keys +(round 4), became perf baselines and now standup prose (round 7) — +same fsynced lines, four products. + +## Omakase notes + +- The check-in is generated *from the artifact of doing the work*, so + it can't be gamed, padded, or forgotten. Status that's a side effect + of working is the only status people don't resent. +- Real deployment: cron this against your production journals Monday + morning, post to the team room. Twenty lines of formatting stand + between this example and that product. + +## Verdict + +The meeting is cancelled and nothing was lost, because the meeting's +only job was extracting what the journal already held. Tools should +answer questions; people should do work. Seven rounds in, this +framework finally has the receipts to enforce that division. diff --git a/examples/weekly_checkin.rb b/examples/weekly_checkin.rb new file mode 100644 index 0000000..57d3b36 --- /dev/null +++ b/examples/weekly_checkin.rb @@ -0,0 +1,84 @@ +# frozen_string_literal: true + +# The Weekly Check-in: "what did you work on this week?" answered by +# the journal instead of by memory. Runs a few days of plans, then +# writes the check-in from the replay - what shipped, what failed, +# where the time went. Nobody's Friday afternoon was harmed. +# +# bundle exec ruby examples/weekly_checkin.rb +# +# Runs offline; the "week" is three journaled plan runs. + +require_relative "../lib/agentic" +require "tmpdir" + +Agentic.logger.level = :fatal + +JOURNAL = File.join(Dir.tmpdir, "agentic_weekly.journal.jsonl") +File.delete(JOURNAL) if File.exist?(JOURNAL) + +journal = Agentic::ExecutionJournal.new(path: JOURNAL) + +# --- the week: three days of plans, journaled as they happen ---------------- +def run_day(journal, day, jobs) + orchestrator = Agentic::PlanOrchestrator.new( + concurrency_limit: 3, + lifecycle_hooks: journal.lifecycle_hooks, + retry_policy: {max_retries: 0, retryable_errors: []} + ) + jobs.each do |name, spec| + orchestrator.add_task(Agentic::Task.new( + description: "#{day}: #{name}", + agent_spec: {"name" => name, "instructions" => "work"}, + payload: spec + ), agent: ->(t) { + sleep(t.payload[:time]) + raise t.payload[:error] if t.payload[:error] + + :done + }) + end + orchestrator.execute_plan +end + +run_day(journal, "mon", { + "import customer csv" => {time: 0.08}, + "dedupe records" => {time: 0.05}, + "sync to billing" => {time: 0.03} +}) +run_day(journal, "wed", { + "backfill invoices" => {time: 0.12}, + "email statements" => {time: 0.02, error: "smtp relay refused"}, + "archive quarter" => {time: 0.04} +}) +run_day(journal, "fri", { + "email statements" => {time: 0.02}, + "close the books" => {time: 0.06} +}) + +# --- the check-in: written from the replay, not from memory ------------------ +state = Agentic::ExecutionJournal.replay(path: JOURNAL) + +shipped = state.completed_descriptions +failures = state.events.select { |e| e[:event] == "task_failed" } +total_time = state.durations.values.sum +slowest = state.durations.max_by { |_, duration| duration } + +puts "WEEKLY CHECK-IN (generated from #{JOURNAL.split("/").last})" +puts +puts "What did you work on this week?" +shipped.each { |item| puts " - #{item}" } +puts +puts "Anything get stuck?" +failures.each do |failure| + recovered = state.completed_descriptions.any? { |d| d.split(": ").last == failure[:description].split(": ").last } + puts " - #{failure[:description]} (#{failure[:error]})" \ + "#{recovered ? " - recovered later in the week" : " - STILL BROKEN"}" +end +puts +puts "Where did the time go?" +puts format(" %.0fms of tracked work; slowest was \"%s\" at %.0fms", + total_time * 1000, slowest[0], slowest[1] * 1000) +puts +puts "written by the journal in 0 minutes. the check-in meeting sends" +puts "its regards, from wherever cancelled meetings go." From 0e8c0b9f5fbf922e5e159cdcc9e5bcf91d66ad06 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 7 Jul 2026 11:19:26 +0000 Subject: [PATCH 28/36] docs: perf history example (tenderlove round 7) This release judged against the journal the last release left behind - description-keyed durations make the durability log double as the performance baseline. Exit 1 on regression against recorded history. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01FeZLdZ3M4q4cho4DZPfSkF --- docs/perspectives/round-7/03-tenderlove.md | 59 ++++++++++++++ examples/perf_history.rb | 90 ++++++++++++++++++++++ 2 files changed, 149 insertions(+) create mode 100644 docs/perspectives/round-7/03-tenderlove.md create mode 100644 examples/perf_history.rb diff --git a/docs/perspectives/round-7/03-tenderlove.md b/docs/perspectives/round-7/03-tenderlove.md new file mode 100644 index 0000000..bb0e929 --- /dev/null +++ b/docs/perspectives/round-7/03-tenderlove.md @@ -0,0 +1,59 @@ +# Round 7 field notes — Aaron Patterson consults the history + +*Built: `examples/perf_history.rb` — this release's run judged against +the journal the last release left behind. The baseline is what +production actually did.* + +## What I built and why + +My round-6 perf diff had one dishonesty I confessed at the time: it +re-ran the baseline in the same process, same load, same thermal +state — a luxury production never gets. The round-7 release closed the +gap: journal replays now expose `durations` keyed by description, so +the baseline is **the fsynced record of the last run that actually +shipped**: + +``` +generate:captions 80ms -> 140ms +60ms REGRESSED +package:episode 50ms -> 20ms -30ms faster +exit 1 +``` + +No benchmark rig, no synthetic load, no "runs on my machine." The +journal was written for crash recovery (round 2); it turns out a +durability log and a performance baseline are the same file read with +different questions. That's my favorite kind of feature — the one +that's been sitting in your data the whole time waiting for an +accessor. + +## Why description-keyed baselines are the right physics + +Task ids are per-run UUIDs; descriptions are stable. That decision — +made by Perham in round 4 for *resume* semantics — is exactly what +makes cross-release comparison sound: `generate:captions` in release 2 +matches `generate:captions` in release 1's journal regardless of when +either ran, in which process, at what id. One naming discipline, three +payoffs (resume, check-ins, baselines). Stable identity is the +cheapest infrastructure you'll ever build. + +Practical notes for real deployments: + +- **Latest-wins is the right default** for the durations map (the + journal may hold many runs; you baseline against the most recent), + but percentile baselines over N runs would resist noise better — + the events array has everything needed; a p50-of-last-5 helper is a + ten-line follow-up. +- The noise floor matters *more* here than in round 6: production + baselines carry production variance. Calibrate against your + journal's own history (stddev per task), not a magic 15ms. +- Missing-from-baseline tasks (new in this release) are skipped, not + flagged — new tasks have no history and get a pass exactly once. + Their first journal entry becomes their accountability. + +## Verdict + +The perf suite's last synthetic component is gone: Gantt, knee, path, +diff — and now the diff's baseline is history instead of hope. CI can +gate releases against the journal of the last one, which means the +question "did we get slower?" finally has a source of truth that +nobody has to maintain. diff --git a/examples/perf_history.rb b/examples/perf_history.rb new file mode 100644 index 0000000..bbaf3cc --- /dev/null +++ b/examples/perf_history.rb @@ -0,0 +1,90 @@ +# frozen_string_literal: true + +# Perf History: last release's run left a journal; this release's run +# is compared against it. No synthetic baseline, no same-process +# rerun - the baseline is what production actually did, keyed by task +# description, straight from ExecutionJournal durations. +# +# bundle exec ruby examples/perf_history.rb +# +# Runs offline; "last release" is journaled first, then "this release" +# runs against its recorded history. + +require_relative "../lib/agentic" +require "tmpdir" + +BASELINE_JOURNAL = File.join(Dir.tmpdir, "agentic_perf_history.jsonl") +File.delete(BASELINE_JOURNAL) if File.exist?(BASELINE_JOURNAL) + +RELEASE_1 = { + "resize:images" => 0.06, + "transcode:audio" => 0.11, + "generate:captions" => 0.08, + "package:episode" => 0.05 +}.freeze + +# This release: captions got slower (new model), packaging got faster +RELEASE_2 = RELEASE_1.merge( + "generate:captions" => 0.14, + "package:episode" => 0.02 +).freeze + +def run_release(work, journal: nil, collect: nil) + hooks = {} + hooks = journal.lifecycle_hooks if journal + if collect + hooks = (journal ? journal.lifecycle_hooks : {}).merge( + after_task_success: ->(task_id:, task:, result:, duration:) { collect[task.description] = duration } + ) + end + + orchestrator = Agentic::PlanOrchestrator.new(concurrency_limit: 4, lifecycle_hooks: hooks) + previous = nil + work.each do |name, latency| + task = Agentic::Task.new(description: name, + agent_spec: {"name" => name, "instructions" => "work"}, payload: latency) + orchestrator.add_task(task, previous ? [previous] : [], agent: ->(t) { sleep(t.payload) || :ok }) + previous = task + end + orchestrator.execute_plan +end + +# --- release 1 ships; its journal IS the baseline ---------------------------- +run_release(RELEASE_1, journal: Agentic::ExecutionJournal.new(path: BASELINE_JOURNAL)) +baseline = Agentic::ExecutionJournal.replay(path: BASELINE_JOURNAL).durations + +# --- release 2 runs; the journal from last time judges it -------------------- +current = {} +run_release(RELEASE_2, collect: current) + +NOISE_MS = 15 + +puts "PERF HISTORY (baseline: #{BASELINE_JOURNAL.split("/").last}, noise floor #{NOISE_MS}ms)" +puts +puts format(" %-20s %10s %10s %9s", "task", "last release", "this one", "delta") +regressions = [] +current.each do |name, duration| + recorded = baseline[name] + next unless recorded + + delta_ms = (duration - recorded) * 1000 + verdict = + if delta_ms.abs < NOISE_MS then "" + elsif delta_ms.negative? then "faster" + else + regressions << name + "REGRESSED" + end + puts format(" %-20s %9.0fms %9.0fms %+8.0fms %s", + name, recorded * 1000, duration * 1000, delta_ms, verdict) +end + +puts +if regressions.any? + puts " #{regressions.join(", ")} regressed against the journal of the" + puts " last shipped release. the baseline wasn't a benchmark rig - it" + puts " was what actually ran, fsynced as it happened." + exit 1 +else + puts " no regressions against recorded history. ship." +end From b6be4864b109a9a1849332304e3293ff4c613f50 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 7 Jul 2026 11:20:43 +0000 Subject: [PATCH 29/36] docs: plan structural diff example (fxn round 7) Two plan wire-format versions diffed as topology - tasks added, edges rewired, labels renamed - so plan review happens at design altitude instead of JSON-line altitude. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01FeZLdZ3M4q4cho4DZPfSkF --- docs/perspectives/round-7/04-fxn.md | 59 +++++++++++++++++++ examples/plan_structural_diff.rb | 87 +++++++++++++++++++++++++++++ 2 files changed, 146 insertions(+) create mode 100644 docs/perspectives/round-7/04-fxn.md create mode 100644 examples/plan_structural_diff.rb diff --git a/docs/perspectives/round-7/04-fxn.md b/docs/perspectives/round-7/04-fxn.md new file mode 100644 index 0000000..b744af7 --- /dev/null +++ b/docs/perspectives/round-7/04-fxn.md @@ -0,0 +1,59 @@ +# Round 7 field notes — Xavier Noria diffs the topology + +*Built: `examples/plan_structural_diff.rb` — two versions of a plan's +wire format, diffed as structure: tasks added, edges rewired, labels +renamed.* + +## What I built and why + +Round 6 made plans serializable; serialized artifacts get committed; +committed artifacts get *diffed* — and a line diff of plan JSON +answers the wrong question. Forty changed lines might mean one task +was inserted. The structural diff answers the right one: + +``` ++ task dedupe entries ++ edge parse entries -> dedupe entries ++ edge dedupe entries -> rank entries +- edge parse entries -> rank entries +``` + +Four facts: dedupe appeared, and ranking was rewired to consume it. +The review conversation this produces — "should ranking consume +deduped candidates instead of raw entries?" — is an *architecture* +question, which is the only kind a plan review should be about. Line +diffs make reviewers audit serialization; structural diffs make them +audit design. + +## Precision notes + +- Edges are keyed by `(from, to)` with the label as the *value*, which + cleanly separates three kinds of change: an edge appearing, an edge + vanishing, and an edge merely *renamed* (`~ label` — the dependency + stayed, its declared purpose changed). A renamed label is a design + decision worth its own diff line; keying edges by + `(from, to, label)` would have misreported it as remove + add. +- The diff operates on the wire format, not on live orchestrators — + deliberately. You diff what you commit; diffing in the artifact's + own vocabulary means the tool works on any two plan.json files from + git history without constructing a single Task. +- What it doesn't detect: task *renames* (fetch feed → fetch feeds + reads as remove + add). Rename detection needs content similarity — + the same reason git's own rename detection is heuristic. Honest + tools state their blind spots; this one's is names. + +## The pattern, now visible across three rounds + +Round 5: generate the artifact (Mermaid). Round 6: prove the +round-trip. Round 7: diff two versions. This is the full lifecycle of +a *format* — render, invert, compare — and it's the same maturation +path every serious format walks (source code, schemas, infrastructure +plans). Plans-as-data is no longer a slogan in this gem; it has the +tooling trilogy. + +## Verdict + +Plan changes are now reviewable at the altitude where design lives. +Next stop, someone wires this into CI to comment the structural diff +on PRs that touch plan.json — the same afternoon-sized step every +round seems to end with. diff --git a/examples/plan_structural_diff.rb b/examples/plan_structural_diff.rb new file mode 100644 index 0000000..06aac80 --- /dev/null +++ b/examples/plan_structural_diff.rb @@ -0,0 +1,87 @@ +# frozen_string_literal: true + +# The Structural Diff: two versions of a plan's wire format, diffed as +# TOPOLOGY - tasks added and removed, edges rewired, labels renamed. +# A line diff of plan JSON tells you bytes changed; this tells you what +# changed about the plan. +# +# bundle exec ruby examples/plan_structural_diff.rb +# +# Runs offline; v1 and v2 are built in-process via the round-trip wire +# format, as they would be loaded from two commits of plan.json. + +require_relative "../lib/agentic" + +def step(name) + Agentic::Task.new(description: name, agent_spec: {"name" => name, "instructions" => "work"}) +end + +def wire(orchestrator) + graph = orchestrator.graph + names = graph[:tasks].transform_values(&:description) + { + "tasks" => graph[:order].map { |id| names[id] }, + "edges" => graph[:edges].map { |e| + {"from" => names[e[:from]], "to" => names[e[:to]], "label" => e[:label]&.to_s} + } + } +end + +# --- version 1: last sprint's pipeline --------------------------------------- +v1 = Agentic::PlanOrchestrator.new +fetch = step("fetch feed") +parse = step("parse entries") +rank = step("rank entries") +publish = step("publish digest") +v1.add_task(fetch) +v1.add_task(parse, [fetch]) +v1.add_task(rank, needs: {entries: parse}) +v1.add_task(publish, [rank]) + +# --- version 2: this sprint's - dedupe added, ranking rewired ---------------- +v2 = Agentic::PlanOrchestrator.new +fetch2 = step("fetch feed") +parse2 = step("parse entries") +dedupe2 = step("dedupe entries") +rank2 = step("rank entries") +publish2 = step("publish digest") +v2.add_task(fetch2) +v2.add_task(parse2, [fetch2]) +v2.add_task(dedupe2, needs: {entries: parse2}) +v2.add_task(rank2, needs: {candidates: dedupe2}) +v2.add_task(publish2, [rank2]) + +# --- the diff: sets of names and labeled edges -------------------------------- +def structural_diff(before, after) + edge_key = ->(e) { [e["from"], e["to"]] } + + before_edges = before["edges"].to_h { |e| [edge_key.call(e), e["label"]] } + after_edges = after["edges"].to_h { |e| [edge_key.call(e), e["label"]] } + + { + tasks_added: after["tasks"] - before["tasks"], + tasks_removed: before["tasks"] - after["tasks"], + edges_added: (after_edges.keys - before_edges.keys), + edges_removed: (before_edges.keys - after_edges.keys), + labels_changed: before_edges.keys.intersection(after_edges.keys) + .reject { |k| before_edges[k] == after_edges[k] } + .map { |k| [k, before_edges[k], after_edges[k]] } + } +end + +diff = structural_diff(wire(v1), wire(v2)) + +puts "PLAN STRUCTURAL DIFF (v1 -> v2)" +puts +diff[:tasks_added].each { |t| puts " + task #{t}" } +diff[:tasks_removed].each { |t| puts " - task #{t}" } +diff[:edges_added].each { |(from, to)| puts " + edge #{from} -> #{to}" } +diff[:edges_removed].each { |(from, to)| puts " - edge #{from} -> #{to}" } +diff[:labels_changed].each { |(from, to), old, new| puts " ~ label #{from} -> #{to}: #{old.inspect} => #{new.inspect}" } + +puts +total = diff.values.sum(&:size) +puts " #{total} structural changes. the review question is no longer" +puts " 'what do these 40 changed JSON lines mean' but 'should ranking" +puts " consume deduped candidates instead of raw entries' - which is" +puts " a question a human can actually answer." From e5c8b44d8fb57c36796159cccf9b2d237c4b7f9d Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 7 Jul 2026 11:22:12 +0000 Subject: [PATCH 30/36] docs: composed limits example (ioquatix round 7) quota.and(pool) under load: both laws hold in their own dimensions and the admission chart names the binding constraint. Ordering contract (quota before socket) demonstrated and explained. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01FeZLdZ3M4q4cho4DZPfSkF --- docs/perspectives/round-7/05-ioquatix.md | 57 +++++++++++++++++++++++ examples/composed_limits.rb | 59 ++++++++++++++++++++++++ 2 files changed, 116 insertions(+) create mode 100644 docs/perspectives/round-7/05-ioquatix.md create mode 100644 examples/composed_limits.rb diff --git a/docs/perspectives/round-7/05-ioquatix.md b/docs/perspectives/round-7/05-ioquatix.md new file mode 100644 index 0000000..d66bf4a --- /dev/null +++ b/docs/perspectives/round-7/05-ioquatix.md @@ -0,0 +1,57 @@ +# Round 7 field notes — Samuel Williams composes the laws + +*Built: `examples/composed_limits.rb` — `quota.and(pool)` (this +round's release) enforcing a billed window and a connection ceiling +simultaneously, with the run showing which law binds.* + +## What I built and why + +Round 6 ended with my note that production APIs enforce two laws at +once and composition was "the user's sentence to write." The release +made it the framework's sentence — `quota.and(pool)` — so this round +characterizes the composition under load: + +``` +window 1: ###### 6 admitted (quota allows 6) +window 2: ###### 6 admitted +pool high-water: 2 of 2 - the socket law held +``` + +Both laws held in their own dimensions, and — the part worth the +example — the run reveals **which one binds**. The pool could clear +~8 calls per window; the quota admits 6; the chart shows exactly 6. +Raise the quota and the wall moves to the pool. Capacity planning is +mostly the art of knowing your binding constraint, and a composed +limiter that *shows* it beats two separate limiters that each swear +they're innocent. + +## The confession, and the lesson it carried + +My first draft's closing prose declared the pool the binding +constraint. The chart said otherwise — 6 per window, quota-bound, +arithmetic I'd done wrong in my head (8 > 6, the pool had slack). I +fixed the prose, not the chart. Seventh round, and the tools are now +routinely correcting their own authors mid-example; I've stopped +counting this as embarrassment and started counting it as the +methodology working. **Write the measurement first, the narrative +second.** + +## Ordering, the subtle contract + +`quota.and(pool)` acquires the quota *before* waiting for a socket. +That order is load-bearing: quota refills on a clock whether you're +ready or not, while sockets are scarce and held. The reverse order +would hold a connection hostage while waiting on the meter — a +convoy generator. The Composite acquires in the order you compose, +so the order you write IS the policy; the docs now say so, and the +example demonstrates the correct idiom. (Composition APIs that make +ordering invisible make deadlocks invisible too — flattening the +nesting but preserving the sequence was the right call in the +implementation.) + +## Verdict + +Two laws, one object, a chart that names the bottleneck, and an +ordering contract stated out loud. The rate-limiting arc that started +with a crowbarred semaphore in round 4 is now a small, honest algebra +— which is all infrastructure ever needed to be. diff --git a/examples/composed_limits.rb b/examples/composed_limits.rb new file mode 100644 index 0000000..49fc348 --- /dev/null +++ b/examples/composed_limits.rb @@ -0,0 +1,59 @@ +# frozen_string_literal: true + +# Composed Limits: a real provider enforces BOTH a billed quota and a +# connection ceiling. quota.and(pool) - new this round - holds the two +# laws in one limiter, and the run proves each law bound the traffic +# in its own dimension: admissions per window AND concurrent in-flight. +# +# bundle exec ruby examples/composed_limits.rb +# +# Runs offline; 12 slow calls against quota 6/200ms and pool of 2. + +require_relative "../lib/agentic" +require "async" + +REQUESTS = 12 +CALL_TIME = 0.05 + +quota = Agentic::RateLimit.new(6, per: 0.2) # billing law: 6 per 200ms +pool = Agentic::RateLimit.new(2) # socket law: 2 in flight +limit = quota.and(pool) # both, in that order + +admissions = [] +started = Process.clock_gettime(Process::CLOCK_MONOTONIC) + +Sync do + REQUESTS.times.map { + Async do + limit.acquire do + admissions << Process.clock_gettime(Process::CLOCK_MONOTONIC) - started + sleep(CALL_TIME) + end + end + }.each(&:wait) +end + +puts "COMPOSED LIMITS: #{REQUESTS} calls x #{(CALL_TIME * 1000).round}ms, " \ + "quota 6/200ms AND pool of 2" +puts + +buckets = admissions.group_by { |t| (t / 0.2).floor } +buckets.sort.each do |window, hits| + puts format(" window %d (%3d-%3dms): %-8s %d admitted (quota allows 6)", + window + 1, window * 200, (window + 1) * 200, "#" * hits.size, hits.size) +end + +puts +puts format(" pool high-water: %d of %d - the socket law held", pool.high_water, pool.ceiling) +puts format(" quota high-water: %d concurrent (window law measures admissions, not flight)", quota.high_water) +puts +puts "which law binds? the pool could clear ~8 per window (2 lanes x 4" +puts "service times) but the quota admits only 6 - so the QUOTA is the" +puts "binding constraint, and the chart shows it: exactly 6 per window." +puts "raise the quota and the pool becomes the wall. composition enforces" +puts "both laws and the windows tell you which one is throttling you." +puts +puts "ordering note: quota.and(pool) spends quota BEFORE waiting for a" +puts "socket - correct, because sockets are scarce and quota refills on a" +puts "clock. the reverse order would hold a connection hostage while" +puts "waiting for the meter." From 267b30ac2d475fd99075d34b59e5cf0db4552b4e Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 7 Jul 2026 11:23:21 +0000 Subject: [PATCH 31/36] docs: backoff conformance example (Jeremy Evans round 7) Every strategy x jitter combination checked against its documented envelope via the new injectable rng - containment AND coverage, a thousand seeded draws each, CI-ready exit code. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01FeZLdZ3M4q4cho4DZPfSkF --- docs/perspectives/round-7/06-jeremyevans.md | 57 ++++++++++++++ examples/backoff_conformance.rb | 85 +++++++++++++++++++++ 2 files changed, 142 insertions(+) create mode 100644 docs/perspectives/round-7/06-jeremyevans.md create mode 100644 examples/backoff_conformance.rb diff --git a/docs/perspectives/round-7/06-jeremyevans.md b/docs/perspectives/round-7/06-jeremyevans.md new file mode 100644 index 0000000..fd9afdd --- /dev/null +++ b/docs/perspectives/round-7/06-jeremyevans.md @@ -0,0 +1,57 @@ +# Round 7 field notes — Jeremy Evans certifies the backoff + +*Built: `examples/backoff_conformance.rb` — every strategy × jitter +combination, a thousand seeded draws each, checked against the +documented envelope. Nine combinations, nine conformances.* + +## What I built and why + +Retry timing is a contract: "exponential with equal jitter" *means* +`[0.75·b·2ⁿ⁻¹, 1.25·b·2ⁿ⁻¹]`, and until this round that meaning was +enforced by nothing but the implementation agreeing with itself. The +`rng:` injection shipped, so the contract became testable without +stubbing a single method — hand the policy a seeded `Random`, collect +a thousand delays per combination, check the envelope: + +``` +exponential true [3.000, 5.000] [3.002, 4.998] conforms +exponential :full [0.000, 4.000] [0.003, 3.995] conforms +``` + +Two properties per combination, because bounds alone are a half-test: + +1. **Containment** — no draw escapes the documented envelope. This is + the safety property; a violation means the docs lie. +2. **Coverage** — the draws *span* at least 80% of the envelope. This + is the liveness property, and it's the one naive tests skip: a + buggy jitter that always returns the midpoint stays in bounds + forever while providing zero herd protection. Perham's stampede + charts are the *reason* for jitter; coverage is the check that the + reason is being served. + +## Why injection beats stubbing, said once more with feeling + +The old way to test this was `allow(orchestrator).to receive(:rand)` +— reaching into an object and replacing its organs. Injection inverts +it: the policy declares "I consume randomness," the test supplies a +seeded source, and *nothing about the object is faked*. The delays +measured here are the delays production would sleep, given that seed. +Tests that exercise real code paths with controlled inputs are worth +ten that exercise fake code paths with real inputs. This is why I ask +every round for effects to become injectable dependencies — clocks +and RNGs first, always. + +## Notes + +- The epsilon (1e-9) on the containment check is float hygiene, not + slack: `rand(0.75..1.25)` can return the boundary. +- This file is CI-shaped: exit 1 on violation, seed in ARGV for + reproduction. It joins the fuzzer, the prober, and the verifier in + what is now a four-tool honesty suite — contracts, rules, docs, and + now *timing*. + +## Verdict + +Nine combinations, nine conformances, two properties each, one seed. +The retry policy's timing promises are no longer folklore backed by +code reading — they're an envelope with a certificate. diff --git a/examples/backoff_conformance.rb b/examples/backoff_conformance.rb new file mode 100644 index 0000000..08de191 --- /dev/null +++ b/examples/backoff_conformance.rb @@ -0,0 +1,85 @@ +# frozen_string_literal: true + +# Backoff Conformance: every strategy x jitter combination, a thousand +# draws each through an injected seeded RNG, checked against the +# documented bounds. Retry timing is a contract like any other - and +# now that rng: is injectable, the contract is testable without +# stubbing a single method. +# +# bundle exec ruby examples/backoff_conformance.rb [seed] +# +# Runs offline and deterministically. Exit 1 on any bound violation. + +require_relative "../lib/agentic" + +seed = (ARGV.first || 20260707).to_i +DRAWS = 1_000 +BASE = 1.0 +RETRY_COUNT = 3 + +# The documented bounds for each (strategy, jitter) combination +def expected_bounds(strategy, jitter, retry_count) + nominal = case strategy + when :constant then BASE + when :linear then retry_count * BASE + when :exponential then BASE * (2**(retry_count - 1)) + end + + case jitter + when false then [nominal, nominal] + when true then [nominal * 0.75, nominal * 1.25] + when :full then [0.0, nominal] + end +end + +failures = [] +puts "BACKOFF CONFORMANCE (seed #{seed}, #{DRAWS} draws per combination)" +puts +puts format(" %-13s %-8s %-22s %-22s %s", "strategy", "jitter", "expected", "observed", "verdict") + +%i[constant linear exponential].each do |strategy| + [false, true, :full].each do |jitter| + rng = Random.new(seed) + orchestrator = Agentic::PlanOrchestrator.new(retry_policy: { + backoff_strategy: strategy, + backoff_constant: BASE, + backoff_base: BASE, + backoff_jitter: jitter, + rng: rng + }) + + observed = [] + orchestrator.define_singleton_method(:sleep) { |delay| observed << delay } + + task = Agentic::Task.new(description: "t", agent_spec: {"name" => "t", "instructions" => "t"}) + task.retry_count = RETRY_COUNT + DRAWS.times { orchestrator.apply_retry_backoff(task: task) } + + low, high = expected_bounds(strategy, jitter, RETRY_COUNT) + epsilon = 1e-9 + in_bounds = observed.all? { |d| d >= low - epsilon && d <= high + epsilon } + spans = jitter == false || (observed.max - observed.min) > (high - low) * 0.8 + + verdict = if !in_bounds + failures << [strategy, jitter, :bounds] + "OUT OF BOUNDS (#{observed.min.round(3)}..#{observed.max.round(3)})" + elsif !spans + failures << [strategy, jitter, :coverage] + "poor coverage" + else + "conforms" + end + + puts format(" %-13s %-8s [%6.3f, %6.3f] [%6.3f, %6.3f] %s", + strategy, jitter.inspect, low, high, observed.min, observed.max, verdict) + end +end + +puts +if failures.empty? + puts " every combination stayed inside its documented envelope AND" + puts " explored at least 80% of it. the timing contract holds." +else + puts " CONTRACT VIOLATIONS: #{failures.inspect}" + exit 1 +end From 5cbb56397d8d4f9df983101e75a2dbc6a2063129 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 7 Jul 2026 11:25:14 +0000 Subject: [PATCH 32/36] docs: JSON Schema export with agreement proof (solnic round 7) The exported draft-07 schema raced against the live validator on 200 seeded payloads (balanced across accept/reject) - zero disagreements. Establishes the house rule: every projection ships its referee. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01FeZLdZ3M4q4cho4DZPfSkF --- docs/perspectives/round-7/07-solnic.md | 63 ++++++++++++ examples/json_schema_export.rb | 131 +++++++++++++++++++++++++ 2 files changed, 194 insertions(+) create mode 100644 docs/perspectives/round-7/07-solnic.md create mode 100644 examples/json_schema_export.rb diff --git a/docs/perspectives/round-7/07-solnic.md b/docs/perspectives/round-7/07-solnic.md new file mode 100644 index 0000000..4a59935 --- /dev/null +++ b/docs/perspectives/round-7/07-solnic.md @@ -0,0 +1,63 @@ +# Round 7 field notes — Piotr Solnica proves the projection + +*Built: `examples/json_schema_export.rb` — a contract exported as +draft-07 JSON Schema, then proven faithful: 200 seeded payloads judged +by the live validator and by an independent interpreter reading only +the exported document. Zero disagreements, 44 accepts, 156 rejects.* + +## What I built and why + +`to_json_schema` shipped this round (my round-6 ask), and an exporter +without a fidelity proof is a rumor generator — the whole value of +exporting your contract to OpenAPI-land is that the *frontend's* +validator and the *backend's* validator enforce the same law. So the +example is two things: the export, and the **agreement proof**. An +independent interpreter (deliberately written against only the JSON +document — it has never seen `CapabilitySpecification`) and Agentic's +own validator judge the same 200 payloads: + +``` +THE AGREEMENT PROOF (200 seeded payloads, 44 valid) + the exported schema and the live validator agreed on every payload. +``` + +Differential testing is the correct verification for any projection: +don't inspect the output, *race it against the source* on shared +inputs. Jeremy's round-6 prober did it for rule declarations; this +does it for the schema export. Between them, a pattern for this +codebase is now established: **every projection ships with its +disagreement detector.** + +## The generator lesson (a confession by proxy) + +The first payload generator produced only 2 valid payloads in 200 — +statistically the proof barely tested the *accept* path, where the +subtle disagreements live (a too-lenient export rejects nothing it +shouldn't; a too-strict one fails only on accepts). The fix: half the +payloads start valid and suffer exactly one corruption. Coverage of +both verdicts is to differential testing what Jeremy's coverage check +was to bounds testing — the half everyone forgets. + +## Notes on the export itself + +- `non_empty` maps to `minLength` for strings and `minItems` for + arrays — the export knows JSON Schema's vocabulary rather than + inventing `x-agentic-non-empty` extensions. Speak the target + language natively or don't export. +- What doesn't project: cross-field `rules:`. JSON Schema's + `dependencies`/`if-then-else` could express *some* of them, but a + lambda is not data, so the export honestly omits them. The API + reference (round 6) documents rules as prose instead — different + projections for different expressiveness, each honest about its + limits. +- `additionalProperties: true` mirrors the validator's + unknown-keys-permitted stance. An exporter that silently tightened + this would fail the agreement proof within ten payloads — which is + precisely why the proof exists. + +## Verdict + +The contract now has five faithful projections — validator, error +payload, reference doc, JSON Schema, and (via Jeremy) an audit — and +the new one arrived with a proof of faithfulness in the same commit. +That should be the house rule: no projection without its referee. diff --git a/examples/json_schema_export.rb b/examples/json_schema_export.rb new file mode 100644 index 0000000..f5a3ffa --- /dev/null +++ b/examples/json_schema_export.rb @@ -0,0 +1,131 @@ +# frozen_string_literal: true + +# The Schema Export: a capability contract emitted as draft-07 JSON +# Schema (new this round), then PROVEN faithful - the same payloads +# are judged by Agentic's validator and by an independent interpreter +# reading only the exported document. Any disagreement means the +# projection lies. 200 seeded payloads: zero disagreements. +# +# bundle exec ruby examples/json_schema_export.rb [seed] +# +# Runs offline; prints the schema and the agreement score. + +require_relative "../lib/agentic" +require "json" + +seed = (ARGV.first || 20260707).to_i +rng = Random.new(seed) + +spec = Agentic::CapabilitySpecification.new( + name: "book_shipment", + description: "Book a shipment", + version: "1.0.0", + inputs: { + mode: {type: "string", required: true, enum: %w[air sea road]}, + weight_kg: {type: "number", required: true, min: 1, max: 30_000}, + reference: {type: "string", required: true, non_empty: true}, + fragile: {type: "boolean"}, + tags: {type: "array", non_empty: true} + } +) + +schema = spec.to_json_schema + +puts "THE EXPORTED SCHEMA" +puts JSON.pretty_generate(schema).gsub(/^/, " ") +puts + +# --- an independent interpreter that knows ONLY the JSON document ------------ +def schema_accepts?(schema, payload) + data = payload.transform_keys(&:to_s) + + return false unless schema["required"].all? { |key| data.key?(key) } + + schema["properties"].all? do |key, rules| + next true unless data.key?(key) + + value = data[key] + type_ok = case rules["type"] + when "string" then value.is_a?(String) + when "number" then value.is_a?(Numeric) + when "boolean" then value == true || value == false + when "array" then value.is_a?(Array) + else true + end + + type_ok && + (rules["enum"].nil? || rules["enum"].include?(value)) && + (rules["minimum"].nil? || (value.is_a?(Numeric) && value >= rules["minimum"])) && + (rules["maximum"].nil? || (value.is_a?(Numeric) && value <= rules["maximum"])) && + (rules["minLength"].nil? || (value.respond_to?(:length) && value.length >= rules["minLength"])) && + (rules["minItems"].nil? || (value.is_a?(Array) && value.size >= rules["minItems"])) + end +end + +# --- generate payloads that wander in and out of validity -------------------- +def valid_payload(rng) + { + mode: %w[air sea road].sample(random: rng), + weight_kg: rng.rand(1..30_000), + reference: "REF-#{rng.rand(1000)}", + fragile: [true, false].sample(random: rng) + } +end + +def corrupted_payload(rng) + payload = valid_payload(rng) + if rng.rand < 0.6 + corruptions = { + mode: ["teleport", 7], weight_kg: [0, "heavy", 50_000], + reference: [""], fragile: ["yes"], tags: [[]] + } + field = corruptions.keys.sample(random: rng) + payload[field] = corruptions[field].sample(random: rng) + end + payload +end + +def chaos_payload(rng) + payload = {} + payload[:mode] = ["air", "sea", "teleport", 7].sample(random: rng) if rng.rand < 0.8 + payload[:weight_kg] = [rng.rand(1..30_000), -4, "heavy"].sample(random: rng) if rng.rand < 0.8 + payload[:reference] = ["REF-#{rng.rand(1000)}", ""].sample(random: rng) if rng.rand < 0.8 + payload[:tags] = [["a"], []].sample(random: rng) if rng.rand < 0.4 + payload +end + +def random_payload(rng) + # Half start valid and get one field corrupted (or not); half are chaos + (rng.rand < 0.5) ? corrupted_payload(rng) : chaos_payload(rng) +end + +validator = Agentic::CapabilityValidator.new(spec) +trials = 200 +disagreements = [] +accepted = 0 + +trials.times do + payload = random_payload(rng) + + agentic_verdict = begin + validator.validate_inputs!(payload) + true + rescue Agentic::Errors::ValidationError + false + end + + schema_verdict = schema_accepts?(schema, payload) + accepted += 1 if agentic_verdict + disagreements << payload if agentic_verdict != schema_verdict +end + +puts "THE AGREEMENT PROOF (#{trials} seeded payloads, #{accepted} valid)" +if disagreements.empty? + puts " the exported schema and the live validator agreed on every payload." + puts " the projection is faithful: what the JSON document promises is" + puts " exactly what the boundary enforces." +else + puts " DISAGREEMENTS (the export lies about the contract):" + disagreements.first(5).each { |payload| puts " - #{payload.inspect}" } + exit 1 +end From b5c2f45c89e921856ba2160b55f37ef1bfe1b20a Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 7 Jul 2026 11:27:33 +0000 Subject: [PATCH 33/36] docs: incident report example (mperham round 7) The 3am report generated from the journal: what ran, root cause with retryability verdict, what never started (plan-vs-record complement), and a resume plan with banked-work accounting. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01FeZLdZ3M4q4cho4DZPfSkF --- docs/perspectives/round-7/08-mperham.md | 58 +++++++++++++++++ examples/incident_report.rb | 83 +++++++++++++++++++++++++ 2 files changed, 141 insertions(+) create mode 100644 docs/perspectives/round-7/08-mperham.md create mode 100644 examples/incident_report.rb diff --git a/docs/perspectives/round-7/08-mperham.md b/docs/perspectives/round-7/08-mperham.md new file mode 100644 index 0000000..a3e5020 --- /dev/null +++ b/docs/perspectives/round-7/08-mperham.md @@ -0,0 +1,58 @@ +# Round 7 field notes — Mike Perham writes the 3am report + +*Built: `examples/incident_report.rb` — a nightly batch dies on +expired credentials; the incident report is generated entirely from +the journal replay.* + +## What I built and why + +Every incident has the same first three questions — what ran, what +broke, what do I resume — and at 3am the difference between a good +night and a bad one is whether answering them requires *grep* or a +*query*. The journal answers all three: + +``` +impact: 3/6 tasks completed before the stop +ROOT CAUSE: load:warehouse - LlmAuthenticationError +completed (do NOT re-run): extract:orders, extract:refunds, transform:ledger +never started (blocked): verify:totals, notify:finance +resume plan: rotate creds -> re-run; 3 journaled tasks skip; ~162ms banked +``` + +Notice what makes the *resume plan* section possible: five rounds of +accumulated journal features composing. Descriptions as idempotency +keys (round 4) → "do NOT re-run" list. Durations (round 7) → "work +already banked." The error *type* on the failure event → the report +can say **"retryable? => false; retrying without fixing creds is +theater"** — the taxonomy from the round-4 drill, now doing incident +triage. The report isn't a template with blanks; every sentence is a +journal fact wearing ops clothes. + +## The design point worth underlining + +The three buckets — completed, failed, never-started — come from set +arithmetic over the replay, and the third bucket is the one dashboards +usually botch. "Never started" isn't in the journal as an event (you +can't journal what didn't happen); it's the *complement* of the plan +against the record. That means the report generator needs the intended +task list plus the journal — plan-as-data (Xavier's wire format) and +journal-as-record are the two halves of every honest incident +timeline. Neither alone can say "verify:totals never ran." + +## Ops notes + +- The failure hook cancels the plan (Jeremy's sentinel pattern) so the + blast radius is bounded and *reported* — cancellation semantics from + round 4 doing exactly what they were fixed for. +- Real deployment: this report is what your pager webhook should + render. The journal is already on disk when the process dies; the + report generator needs nothing from the crashed process's memory. + That's the whole point of fsync-per-event, cashing out four rounds + later. + +## Verdict + +The journal now covers the full ops lifecycle: survive the crash +(round 2), resume by name (round 4), explain the week (this round's +check-in), and brief the incident channel. One JSONL file, four +tools, zero archaeology at 3am. diff --git a/examples/incident_report.rb b/examples/incident_report.rb new file mode 100644 index 0000000..2ec4026 --- /dev/null +++ b/examples/incident_report.rb @@ -0,0 +1,83 @@ +# frozen_string_literal: true + +# The Incident Report: a nightly batch dies at 3am. The on-call's +# first three questions - what ran? what broke? what do I resume? - +# answered from the journal replay, formatted for the incident +# channel. Nobody greps logs at 3am if the journal can already speak. +# +# bundle exec ruby examples/incident_report.rb +# +# Runs offline; the outage is scripted. + +require_relative "../lib/agentic" +require "tmpdir" + +Agentic.logger.level = :fatal + +JOURNAL = File.join(Dir.tmpdir, "agentic_incident.journal.jsonl") +File.delete(JOURNAL) if File.exist?(JOURNAL) + +journal = Agentic::ExecutionJournal.new(path: JOURNAL) + +NIGHTLY = { + "extract:orders" => {time: 0.05}, + "extract:refunds" => {time: 0.04}, + "transform:ledger" => {time: 0.07, deps: %w[extract:orders extract:refunds]}, + "load:warehouse" => {time: 0.03, deps: %w[transform:ledger], + error: Agentic::Errors::LlmAuthenticationError.new("warehouse credentials expired")}, + "verify:totals" => {time: 0.02, deps: %w[load:warehouse]}, + "notify:finance" => {time: 0.01, deps: %w[verify:totals]} +}.freeze + +orchestrator = nil +hooks = journal.lifecycle_hooks( + after_task_failure: ->(task_id:, task:, failure:, duration:) { orchestrator.cancel_plan } +) +orchestrator = Agentic::PlanOrchestrator.new( + concurrency_limit: 2, lifecycle_hooks: hooks, + retry_policy: {max_retries: 0, retryable_errors: []} +) + +tasks = {} +NIGHTLY.each do |name, spec| + tasks[name] = Agentic::Task.new(description: name, + agent_spec: {"name" => name, "instructions" => "run"}, payload: spec) + orchestrator.add_task(tasks[name], (spec[:deps] || []).map { |d| tasks.fetch(d) }, agent: ->(t) { + sleep(t.payload[:time]) + raise t.payload[:error] if t.payload[:error] + + :ok + }) +end +orchestrator.execute_plan + +# --- the report: everything below reads ONLY the journal --------------------- +state = Agentic::ExecutionJournal.replay(path: JOURNAL) +all_tasks = NIGHTLY.keys +completed = state.completed_descriptions +failed = state.events.select { |e| e[:event] == "task_failed" } +never_ran = all_tasks - completed - failed.map { |f| f[:description] } + +puts "INCIDENT REPORT - nightly batch" +puts "=" * 52 +puts +puts "impact:" +puts " #{completed.size}/#{all_tasks.size} tasks completed before the stop" +failed.each do |f| + puts " ROOT CAUSE: #{f[:description]} - #{f[:error_type]}" + puts " \"#{f[:error]}\"" +end +puts +puts "completed (do NOT re-run - outputs are journaled):" +completed.each { |d| puts format(" + %-20s %4.0fms", d, (state.durations[d] || 0) * 1000) } +puts +puts "never started (blocked behind the failure):" +never_ran.each { |d| puts " . #{d}" } +puts +puts "resume plan:" +puts " 1. rotate the warehouse credentials (error is LlmAuthenticationError:" +puts " retryable? => false; retrying without fixing creds is theater)" +puts " 2. re-run the batch - completed?(description) will skip the" +puts " #{completed.size} journaled tasks; only #{all_tasks.size - completed.size} run" +puts format(" 3. budget: ~%.0fms of work already banked, don't pay twice", + state.durations.values.sum * 1000) From 9cb59ed4c5017d3eef01bac7bc6f893309517c43 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 7 Jul 2026 11:28:48 +0000 Subject: [PATCH 34/36] docs: graph style guide example (Sandi Metz round 7) RuboCop for plans: four ten-line cops over graph[:stats] with thresholds and reasons, including NamedFanIns - joins of two or more must name their dependencies. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01FeZLdZ3M4q4cho4DZPfSkF --- docs/perspectives/round-7/09-sandimetz.md | 57 ++++++++++++++ examples/graph_style.rb | 95 +++++++++++++++++++++++ 2 files changed, 152 insertions(+) create mode 100644 docs/perspectives/round-7/09-sandimetz.md create mode 100644 examples/graph_style.rb diff --git a/docs/perspectives/round-7/09-sandimetz.md b/docs/perspectives/round-7/09-sandimetz.md new file mode 100644 index 0000000..3cb3522 --- /dev/null +++ b/docs/perspectives/round-7/09-sandimetz.md @@ -0,0 +1,57 @@ +# Round 7 field notes — Sandi Metz writes the style guide + +*Built: `examples/graph_style.rb` — RuboCop for plans: four cops with +thresholds and reasons, run against a tidy graph and a messy one.* + +## What I built and why + +My critic (round 4) reviewed one plan; my receipts (round 6) tracked +one refactoring. A team needs neither — it needs a *style guide*, +because style guides argue once, in a config, instead of in every +review. Four cops, each with a threshold and — this part is not +optional — a **why**: + +``` +Graph/MaxDepth: present sits 6 deep (limit 4) +Graph/NamedFanIns: funnel joins 4 but 4 edges are unnamed - use needs: + (a join you can't name is a join you don't understand) +``` + +`NamedFanIns` is the cop I built this example for. It lints +*vocabulary*: any join of two or more dependencies must name them via +`needs:`. The rationale is the same one I give for keyword arguments +over positional — at arity two, order starts lying to you — but +here the names do double duty, because Xavier showed us (round 5) +that `needs:` labels become documentation and diagram edges. A +style rule that improves comprehension AND tooling output is the +kind you can actually get a team to adopt. Note the messy plan's +funnel commits two offenses at once: too wide AND unnamed. Wide +anonymous joins are how plans become haunted houses. + +## On thresholds, honestly + +MaxDepth 4 and MaxFanIn 3 are *this example's* taste, not laws — the +closing line says so out loud. The failure mode of every style guide +is thresholds mistaken for physics; the value is never the number, +it's that the number is **written down**, so disagreement becomes a +one-line diff and a conversation instead of a review-thread war. (My +own rules — five lines, four arguments — were always conversation +starters wearing the costume of commandments. It said so in the book; +nobody reads that page.) + +## Framework note + +Every cop reads precomputed facts: `stats[:max_depth]`, +`stats[:depth]`, edge labels. Since `graph[:stats]` shipped (my +third-strike ask, round 7 release), a cop is now a *predicate plus +prose* — ten lines each. When adding a lint rule costs ten lines, +teams write their own; when it costs a graph traversal, they don't. +API convenience isn't a luxury; it's the difference between a tool +and an ecosystem. + +## Verdict + +Critic, receipts, style guide — see the smell, fix the smell, prevent +the smell. The trilogy is complete, each layer cheaper than the one +before it, which is the correct direction: prevention should always +be the cheapest. diff --git a/examples/graph_style.rb b/examples/graph_style.rb new file mode 100644 index 0000000..e484c62 --- /dev/null +++ b/examples/graph_style.rb @@ -0,0 +1,95 @@ +# frozen_string_literal: true + +# The Graph Style Guide: RuboCop for plans. Cops with thresholds run +# against any orchestrator's graph - depth, fan-in, orphans, and the +# style rule I care most about: fan-ins of two or more should NAME +# their dependencies, because a join you can't name is a join you +# don't understand. +# +# bundle exec ruby examples/graph_style.rb +# +# Runs offline; lints a tidy plan and a messy one. + +require_relative "../lib/agentic" + +STYLE = { + "Graph/MaxDepth" => {limit: 4, why: "every level is latency and a failure domain"}, + "Graph/MaxFanIn" => {limit: 3, why: "wide joins own too many failure modes"}, + "Graph/NoOrphans" => {why: "unconnected tasks are in the wrong plan or missing an edge"}, + "Graph/NamedFanIns" => {min_to_name: 2, why: "a join you can't name is a join you don't understand"} +}.freeze + +def lint(graph, style) + names = graph[:tasks].transform_values(&:description) + stats = graph[:stats] + offenses = [] + + if stats[:max_depth] > style["Graph/MaxDepth"][:limit] + deepest = stats[:depth].max_by { |_, d| d }.first + offenses << ["Graph/MaxDepth", "#{names[deepest]} sits #{stats[:max_depth]} deep (limit #{style["Graph/MaxDepth"][:limit]})"] + end + + graph[:dependencies].each do |id, deps| + if deps.size > style["Graph/MaxFanIn"][:limit] + offenses << ["Graph/MaxFanIn", "#{names[id]} joins #{deps.size} (limit #{style["Graph/MaxFanIn"][:limit]})"] + end + end + + graph[:dependencies].each do |id, deps| + if deps.empty? && graph[:edges].none? { |e| e[:from] == id } && graph[:tasks].size > 1 + offenses << ["Graph/NoOrphans", "#{names[id]} touches nothing and is touched by nothing"] + end + end + + graph[:dependencies].each do |id, deps| + next if deps.size < style["Graph/NamedFanIns"][:min_to_name] + + unnamed = graph[:edges].count { |e| e[:to] == id && e[:label].nil? } + if unnamed.positive? + offenses << ["Graph/NamedFanIns", "#{names[id]} joins #{deps.size} but #{unnamed} edge(s) are unnamed - use needs:"] + end + end + + offenses +end + +def step(name) + Agentic::Task.new(description: name, agent_spec: {"name" => name, "instructions" => "work"}) +end + +# --- a tidy plan ------------------------------------------------------------ +tidy = Agentic::PlanOrchestrator.new +a = step("fetch users") +b = step("fetch orders") +c = step("merge report") +tidy.add_task(a) +tidy.add_task(b) +tidy.add_task(c, needs: {users: a, orders: b}) + +# --- a messy one -------------------------------------------------------------- +messy = Agentic::PlanOrchestrator.new +sources = 4.times.map { |i| step("source-#{i}") } +funnel = step("funnel") +steps = %w[polish buff shine present].map { |n| step(n) } +stray = step("stray") +sources.each { |t| messy.add_task(t) } +messy.add_task(funnel, sources) +previous = funnel +steps.each { |t| messy.add_task(t, [previous]) && (previous = t) } +messy.add_task(stray) + +puts "GRAPH STYLE GUIDE (#{STYLE.size} cops)" +{"tidy plan" => tidy, "messy plan" => messy}.each do |label, orchestrator| + offenses = lint(orchestrator.graph, STYLE) + puts + puts " #{label}: #{offenses.empty? ? "no offenses" : "#{offenses.size} offense(s)"}" + offenses.each do |cop, message| + puts " #{cop}: #{message}" + puts " (#{STYLE[cop][:why]})" + end +end + +puts +puts "style guides work because they argue once, in a config file," +puts "instead of every review. these thresholds are this team's taste -" +puts "yours may differ. that they're WRITTEN DOWN is the feature." From 2ece04ed50473921032550d4d1b17b1f096c022e Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 7 Jul 2026 11:30:07 +0000 Subject: [PATCH 35/36] docs: capability evals example (ankane round 7) Golden test cases scored and gated against registered capabilities: a contract-valid but substance-wrong parser caught at 33%. Contracts check types; evals check truth. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01FeZLdZ3M4q4cho4DZPfSkF --- docs/perspectives/round-7/10-ankane.md | 60 ++++++++++++++++ examples/capability_evals.rb | 94 ++++++++++++++++++++++++++ 2 files changed, 154 insertions(+) create mode 100644 docs/perspectives/round-7/10-ankane.md create mode 100644 examples/capability_evals.rb diff --git a/docs/perspectives/round-7/10-ankane.md b/docs/perspectives/round-7/10-ankane.md new file mode 100644 index 0000000..6e75105 --- /dev/null +++ b/docs/perspectives/round-7/10-ankane.md @@ -0,0 +1,60 @@ +# Round 7 field notes — Andrew Kane runs the evals + +*Built: `examples/capability_evals.rb` — golden test cases against +registered capabilities, scored and gated. One capability has a bug +the contracts can't see; the evals catch it.* + +## What I built and why + +Seven rounds of contract machinery check that outputs have the right +*shape*. Evals check the right *substance* — and the seeded bug shows +why both layers exist: + +``` +extract_amount 1/3 (33%) BELOW THRESHOLD + FAIL "you owe $12.50 by friday" + expected {:cents=>1250}, got {:cents=>1200} +``` + +`{cents: 1200}` is a perfectly contract-valid answer — number, present, +positive. It's also *wrong by fifty cents*, because the parser drops +the decimal. No type system catches a plausible lie; only a golden +case does. That's the whole ML-in-production lesson compressed: +**contracts check types, evals check truth, and the failures that hurt +are always in the gap between them.** + +This matters double for this gem specifically: every capability here +is designed to be swapped from lambda to LLM (it's been the pitch +since my round-2 notes). The eval suite is what makes that swap +*safe* — change the implementation, rerun the goldens, read the score. +Same cases, new brain, numeric verdict. Nobody should ship a model +swap on vibes. + +## Design choices worth stealing + +- **Evals live next to the capability name, not the implementation** — + the suite is a property of the *interface*, so it survives every + implementation change. That's also why the expected values are + subsets (`expect.all? { actual[key] == expected }`): implementations + may return extra keys; goldens pin only what's promised. +- Per-capability pass rates plus a suite score with a threshold and + exit code — the shape CI wants. classify_sentiment at 100% doesn't + excuse extract_amount at 33%; averages hide, per-capability tables + testify. +- For LLM-backed capabilities the equality check becomes a scorer + (exact / contains / judge-model), but the harness shape doesn't + change. Start exact, loosen deliberately. + +## The honesty-suite census, seven rounds in + +Fuzzer (types), prober (rule declarations), verifier (docs), +conformance (timing), agreement proof (schema export), and now evals +(substance). Six referee tools, all exit-code-gated, all built on the +gem's own primitives. The framework can no longer lie to you about +its contracts, its docs, its timing, or — as of today — its answers. + +## Verdict + +The gap between "valid" and "correct" now has a test suite. That's +the last gap that matters, and it's the one every team skips until +the fifty-cent bugs compound into a refund queue. diff --git a/examples/capability_evals.rb b/examples/capability_evals.rb new file mode 100644 index 0000000..20f8efb --- /dev/null +++ b/examples/capability_evals.rb @@ -0,0 +1,94 @@ +# frozen_string_literal: true + +# Capability Evals: golden test cases run against registered +# capabilities, scored, and gated. When you swap a lambda for an LLM +# (or one model for another), the eval suite is what tells you whether +# quality moved - contracts check shape, evals check SUBSTANCE. +# +# bundle exec ruby examples/capability_evals.rb +# +# Runs offline; one capability has a bug the evals catch. + +require_relative "../lib/agentic" + +def capability(name, inputs:, outputs:, &impl) + spec = Agentic::CapabilitySpecification.new( + name: name, description: name, version: "1.0.0", inputs: inputs, outputs: outputs + ) + Agentic.register_capability( + spec, Agentic::CapabilityProvider.new(capability: spec, implementation: impl) + ) +end + +capability("classify_sentiment", + inputs: {text: {type: "string", required: true}}, + outputs: {label: {type: "string", required: true, enum: %w[positive negative neutral]}}) do |i| + text = i[:text].downcase + label = if text.match?(/love|great|excellent|happy/) + "positive" + elsif text.match?(/hate|terrible|awful|angry/) + "negative" + else + "neutral" + end + {label: label} +end + +capability("extract_amount", + inputs: {text: {type: "string", required: true}}, + outputs: {cents: {type: "number", required: true}}) do |i| + # The bug: parses dollars but drops the cents + dollars = i[:text][/\$(\d+)/, 1].to_i + {cents: dollars * 100} +end + +EVALS = { + "classify_sentiment" => [ + {input: {text: "I love this gem"}, expect: {label: "positive"}}, + {input: {text: "This is terrible"}, expect: {label: "negative"}}, + {input: {text: "It runs on Ruby 3.3"}, expect: {label: "neutral"}}, + {input: {text: "absolutely EXCELLENT work"}, expect: {label: "positive"}} + ], + "extract_amount" => [ + {input: {text: "invoice total $45"}, expect: {cents: 4500}}, + {input: {text: "you owe $12.50 by friday"}, expect: {cents: 1250}}, + {input: {text: "refund of $0.99 issued"}, expect: {cents: 99}} + ] +}.freeze + +registry = Agentic::AgentCapabilityRegistry.instance +THRESHOLD = 0.9 + +puts "CAPABILITY EVALS (pass threshold #{(THRESHOLD * 100).round}%)" +puts + +overall = [] +EVALS.each do |name, cases| + provider = registry.get_provider(name) + results = cases.map do |eval_case| + actual = provider.execute(eval_case[:input]) + passed = eval_case[:expect].all? { |key, expected| actual[key] == expected } + [eval_case, actual, passed] + end + + passed = results.count { |_, _, ok| ok } + rate = passed.to_f / cases.size + overall << rate + puts format(" %-20s %d/%d (%3.0f%%) %s", name, passed, cases.size, rate * 100, + (rate >= THRESHOLD) ? "" : "BELOW THRESHOLD") + + results.reject { |_, _, ok| ok }.each do |eval_case, actual, _| + puts " FAIL #{eval_case[:input][:text].inspect}" + puts " expected #{eval_case[:expect].inspect}, got #{actual.inspect}" + end +end + +puts +score = overall.sum / overall.size +puts format(" suite score: %.0f%%", score * 100) +if score < THRESHOLD + puts " the gate holds: extract_amount drops cents - a shape-valid," + puts " substance-wrong answer that no contract could catch. contracts" + puts " check types; evals check truth. you need both." + exit 1 +end From 70d3a83fb1882ff6ffe4cf10b7d299063f77e87c Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 7 Jul 2026 11:32:09 +0000 Subject: [PATCH 36/36] docs: round-7 index and findings; regenerate examples catalog Adds the round-7 table to the perspectives README: the referee round - six exit-code honesty tools, the journal as four products, and the next asks. Catalog regenerated at 60 entries. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01FeZLdZ3M4q4cho4DZPfSkF --- docs/perspectives/README.md | 39 +++++++++++++++++++++++++++++++++++++ examples/README.md | 10 ++++++++++ 2 files changed, 49 insertions(+) diff --git a/docs/perspectives/README.md b/docs/perspectives/README.md index 68099b9..4a6959b 100644 --- a/docs/perspectives/README.md +++ b/docs/perspectives/README.md @@ -127,6 +127,45 @@ modernized onto them, and ten more experiments followed: | 9 | Sandi Metz | Refactor receipts — the god join dissolved in priced steps | `examples/refactor_receipts.rb` | [round-6/09-sandimetz.md](round-6/09-sandimetz.md) | | 10 | Andrew Kane | Cost estimator — the plan priced before it runs, reconciled after | `examples/cost_estimator.rb` | [round-6/10-ankane.md](round-6/10-ankane.md) | +## Round 7 — the referee round + +The round-6 asks shipped as a release (`RateLimit#and` composition, +`graph[:stats]`, journal `durations` keyed by description, +`CapabilitySpecification#to_json_schema`, injectable retry `rng:`), +two graph tools were modernized onto `stats`, and ten more experiments +followed: + +| # | Persona | Built on the round-7 release | Run it | Field notes | +|---|---------|------------------------------|--------|-------------| +| 1 | Matz | Plan fortune teller — structural diagnoses in a mystic's robe | `examples/plan_fortune.rb` | [round-7/01-matz.md](round-7/01-matz.md) | +| 2 | DHH | Weekly check-in — the journal answers so nobody's Friday has to | `examples/weekly_checkin.rb` | [round-7/02-dhh.md](round-7/02-dhh.md) | +| 3 | Aaron Patterson | Perf history — regressions judged against the last release's journal | `examples/perf_history.rb` | [round-7/03-tenderlove.md](round-7/03-tenderlove.md) | +| 4 | Xavier Noria | Structural diff — plan review at design altitude, not JSON altitude | `examples/plan_structural_diff.rb` | [round-7/04-fxn.md](round-7/04-fxn.md) | +| 5 | Samuel Williams | Composed limits — both laws at once; the chart names the binding one | `examples/composed_limits.rb` | [round-7/05-ioquatix.md](round-7/05-ioquatix.md) | +| 6 | Jeremy Evans | Backoff conformance — nine timing envelopes certified via injected rng | `examples/backoff_conformance.rb` | [round-7/06-jeremyevans.md](round-7/06-jeremyevans.md) | +| 7 | Piotr Solnica | Schema export + agreement proof — every projection ships its referee | `examples/json_schema_export.rb` | [round-7/07-solnic.md](round-7/07-solnic.md) | +| 8 | Mike Perham | Incident report — the 3am questions answered from the journal | `examples/incident_report.rb` | [round-7/08-mperham.md](round-7/08-mperham.md) | +| 9 | Sandi Metz | Graph style guide — RuboCop for plans, ten lines per cop | `examples/graph_style.rb` | [round-7/09-sandimetz.md](round-7/09-sandimetz.md) | +| 10 | Andrew Kane | Capability evals — contracts check types, evals check truth | `examples/capability_evals.rb` | [round-7/10-ankane.md](round-7/10-ankane.md) | + +### What round 7 surfaced + +1. **The referee pattern generalized**: six exit-code-gated honesty + tools now exist (fuzzer, prober, verifier, conformance, agreement + proof, evals) — the framework can no longer lie about its + contracts, rules, docs, timing, exports, or answers. +2. **The journal became four products**: crash recovery, resume keys, + perf baselines, and prose (check-ins, incident reports) — one + fsynced JSONL file, read with different questions. +3. **Tools kept correcting their authors**: Samuel's binding-constraint + prose and Piotr's generator coverage were both fixed by their own + measurements — the third straight round of measurement-over-narrative. +4. **Next asks**: `stats[:roots]`/`stats[:leaves]`, percentile + baselines over journal history (p50-of-last-N), rename detection + hints in the structural diff, JSON Schema `if/then` emission for + expressible rules, and an eval-scorer seam for LLM-backed + capabilities. + ### What round 6 surfaced 1. **Plans became artifacts**: narratable (tour), serializable with an diff --git a/examples/README.md b/examples/README.md index c02c1aa..b4c16df 100644 --- a/examples/README.md +++ b/examples/README.md @@ -7,10 +7,13 @@ not this file. | Example | What it shows | |---------|---------------| | `api_reference.rb` | The API Reference Generator: walk the registry, emit reference docs for every capability - types, enums, bounds, policie... | +| `backoff_conformance.rb` | Backoff Conformance: every strategy x jitter combination, a thousand draws each through an injected seeded RNG, checked ... | | `burst_absorber.rb` | The Burst Absorber: three waves of requests slam a credential with a ceiling of 3 (Agentic::RateLimit - this round's rel... | +| `capability_evals.rb` | Capability Evals: golden test cases run against registered capabilities, scored, and gated. When you swap a lambda for a... | | `changelog_scout.rb` | The Changelog Scout: reads real git history, classifies every commit through a contract-checked capability, and drafts t... | | `collaboration_tracer.rb` | The Collaboration Tracer: lifecycle hooks record every message the orchestrator sends and every reply that comes back, t... | | `command_bus.rb` | The Command Bus: every command is a composed capability with its OWN declared contract (new in this round - compositions... | +| `composed_limits.rb` | Composed Limits: a real provider enforces BOTH a billed quota and a connection ceiling. quota.and(pool) - new this round... | | `contract_fuzzer.rb` | The Contract Fuzzer: for every registered capability, generate inputs that SHOULD pass its declared contract and mutatio... | | `cost_estimator.rb` | The Cost Estimator: price the plan BEFORE running it - per-task token estimates times a pricing table - gate on budget, ... | | `coupling_cartographer.rb` | The Coupling Cartographer: which files lean on which? Every file is surveyed for the constants it DEFINES and the consta... | @@ -26,19 +29,25 @@ not this file. | `freight_rules.rb` | The Freight Desk: a quoting capability whose tariff book is written as cross-field contract rules (new this round). Per-... | | `gem_scout.rb` | Gem Scout: describe what you need, get a ranked shortlist of gems. Search and scoring are separate capabilities; the sea... | | `graph_critic.rb` | The Graph Critic: reviews a plan's dependency structure BEFORE it runs, the way you'd review a class diagram. God tasks,... | +| `graph_style.rb` | The Graph Style Guide: RuboCop for plans. Cops with thresholds run against any orchestrator's graph - depth, fan-in, orp... | | `haiku_agent.rb` | The three-line agent. Run me with no API key at all: | +| `incident_report.rb` | The Incident Report: a nightly batch dies at 3am. The on-call's first three questions - what ran? what broke? what do I ... | | `invariant_sentinel.rb` | The Invariant Sentinel: domain invariants checked after EVERY task, from a lifecycle hook. When a task leaves the world ... | | `jitter_shootout.rb` | The Jitter Shootout: none vs equal (+/-25%, the default) vs full (uniform over [0, delay], new this round) - same forty ... | +| `json_schema_export.rb` | The Schema Export: a capability contract emitted as draft-07 JSON Schema (new this round), then PROVEN faithful - the sa... | | `kanban_board.rb` | The Kanban Board: a plan rendered as the three columns everyone actually understands - To Do, Doing, Done - reprinted at... | | `knee_finder.rb` | The Knee Finder: runs the same plan at increasing concurrency limits, measures wall time and total queue-wait via the ta... | | `latency_lab.rb` | The Latency Lab: 20 simulated LLM calls (200ms of IO each) executed through the orchestrator at different concurrency li... | | `live_dashboard.rb` | The Live Dashboard: lifecycle hooks publish events onto an Async::Queue; a consumer task IN THE SAME REACTOR renders the... | | `namespace_cartographer.rb` | The Namespace Cartographer: maps a gem's constant tree and audits every file against the constant Zeitwerk expects it to... | | `perf_diff.rb` | The Perf Diff: run the plan before and after a change, diff per-task durations, and flag regressions - with the one qual... | +| `perf_history.rb` | Perf History: last release's run left a journal; this release's run is compared against it. No synthetic baseline, no sa... | | `performance_detective.rb` | The Performance Detective: one task per Ruby file in lib/, fanned out through the orchestrator, each dissecting a file f... | | `plan_diagram.rb` | The Plan Diagrammer: any orchestrator's graph, emitted as Mermaid - paste it into a README, GitHub renders it, and the d... | +| `plan_fortune.rb` | The Plan Fortune Teller: reads your graph's palm - depth, fan-in, roots, breadth - and tells its fortune. Every fortune ... | | `plan_gantt.rb` | The Plan Gantt: lifecycle hooks timestamp every task, then the run is rendered as an ASCII timeline - where your wall cl... | | `plan_roundtrip.rb` | The Round Trip: serialize a plan's graph to JSON, rebuild a fresh orchestrator from the JSON, and prove the rebuilt topo... | +| `plan_structural_diff.rb` | The Structural Diff: two versions of a plan's wire format, diffed as TOPOLOGY - tasks added and removed, edges rewired, ... | | `plan_tour.rb` | The Plan Tour: hand any orchestrator to the guide and it narrates the plan as prose - first this, then that, meanwhile t... | | `quota_keeper.rb` | The Quota Keeper: the same 20 requests through two different laws. A concurrency ceiling ("3 in flight") models connecti... | | `readme_verifier.rb` | The README Verifier: every ruby code fence in the README is a promise. This extracts them all, syntax-checks each with P... | @@ -56,3 +65,4 @@ not this file. | `three_shapes.rb` | Three Shapes: the same six units of work arranged three ways - a chain, a star, and staged joins - then measured and cri... | | `ticket_screener.rb` | A HEY-style ticket screener: every inbound support ticket flows through screen -> categorize -> draft, all tickets in pa... | | `typed_pipeline.rb` | A typed ETL pipeline: extract -> transform -> load, each stage a capability with a declared contract, composed into one ... | +| `weekly_checkin.rb` | The Weekly Check-in: "what did you work on this week?" answered by the journal instead of by memory. Runs a few days of ... |