Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
65 changes: 65 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,71 @@

All notable changes to blockrun-llm will be documented in this file.

## 1.11.0 — 2026-08-15

### Added
- **Router Core lands in the Python SDK.** `blockrun_llm/router_core/` is a
faithful port of [`@blockrun/router-core`](https://github.com/BlockRunAI/router-core)
(upstream commit `18bf4ab`) — the product-neutral routing engine the
TypeScript SDK bundles and the gateway runs. The same request now routes
identically across all three. `blockrun_llm/router_adapter.py` is the host
glue (catalog id resolution, x402 payment floors, capacity filtering), ported
from the TypeScript SDK's `src/router-adapter.ts`.

What the Python SDK did not have before:
- **Portfolio (V3) ranking**, not just tier lookup: candidates are scored on
task affinity, cost, speed and reliability, so the cheapest *capable* model
wins instead of a hardcoded tier primary.
- **Hard capability filtering.** A model that cannot hold the conversation,
emit the requested `max_tokens`, call tools, or read images is dropped
before scoring — previously `smart_chat` could route to a model the request
would fail on with a non-transient 400.
- **Task classification** (`chat`, `code_edit`, `code_agent`, `tool_agent`,
`tool_agent_parallel`, `reasoning_math`, `reasoning_mcq`, `long_context`,
`extraction`, `vision`, `debug`) with per-task calibrated model evidence.
- **Explainable decisions**: `routing.candidates`, `routing.candidate_scores`
(quality / cost / speed / reliability per model), `routing.task_type`,
`routing.profile` and `routing.router_version` are now on the response.
- **Live tier configuration**, shared with the other products, replacing this
SDK's separately hand-maintained tables.

- **`client.route(prompt, ...)`** returns the routing decision without making or
paying for a model call (TypeScript SDK parity). The first call may fetch the
public catalog for prices; routing itself is local and free.

### Fixed
- **The `free` profile pointed at models NVIDIA has retired.** Its tier table
led with `nvidia/deepseek-v4-flash` (EOL 2026-08-12, HTTP 410) and fell back
to `nvidia/llama-4-maverick` and `nvidia/qwen3-coder-480b` (also EOL), so free
routing depended entirely on the gateway's redirect safety net. It now routes
over the live free lineup (Step 3.7 Flash, Mistral Nemotron, Nemotron Nano
Omni / 9B / 12B VL), and the adapter drops any candidate the catalog does not
price at $0 — a paid model can no longer leak into a free-profile call.
- **Models the catalog marks unavailable no longer win routing.** `/v1/models`
rows with `available: false` are skipped when building the pricing map; every
smart call to one would have failed with a non-transient error.

### Changed
- `routing.method` is now `"portfolio"` for the default strategy (`"rules"` for
the free profile and the config-only V2 rollback). Code that asserted
`method == "rules"` needs updating.
- `blockrun_llm/router.py` is now a thin compatibility shim over the core:
`route()` and `classify_by_rules()` keep working, and `RoutingDecision` keeps
its previous keys plus the new metadata. Its hand-maintained `AUTO_TIERS` /
`ECO_TIERS` / `PREMIUM_TIERS` tables are gone — tier configuration lives in
`router_core.DEFAULT_ROUTING_CONFIG`, and `FREE_TIERS` moved to
`router_adapter`.
- Routing cost estimates now include the server margin and the x402 minimum
payment, so `routing.cost_estimate` matches what the gateway actually
charges. Free models are never floored up to the paid minimum.

### Tests
- `tests/unit/test_router_core.py` ports all four upstream vitest suites
(88 cases) as the parity guard — the Python port must keep choosing the same
models as the TypeScript SDK. `tests/unit/test_router_adapter.py` covers the
host layer: `free/*` → `nvidia/*` id resolution, the payment floor, capacity
filtering, and the free-profile guarantees.

## 1.10.0 — 2026-07-28

### Added
Expand Down
4 changes: 3 additions & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,9 @@ blockrun_llm/
├── wallet.py # EVM wallet management
├── solana_wallet.py # Solana wallet management
├── x402.py # x402 payment protocol
├── router.py # Model routing
├── router_core/ # Port of @blockrun/router-core (shared with the TS SDK + gateway)
├── router_adapter.py # Host glue: catalog ids, payment floors, free profile
├── router.py # Back-compat shim over router_core
├── types.py # Type definitions
├── validation.py # Input validation
├── cache.py # Response caching
Expand Down
84 changes: 57 additions & 27 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -121,7 +121,7 @@ export SOLANA_WALLET_KEY="your-bs58-solana-key"
> what to switch to instead of failing with a cryptic "must be 66 characters"
> error.

## Smart Routing (ClawRouter)
## Smart Routing (Router Core)

Let the SDK automatically pick the cheapest capable model for each request:

Expand All @@ -130,54 +130,84 @@ from blockrun_llm import LLMClient

client = LLMClient()

# Auto-routes to cheapest capable model
result = client.smart_chat("What is 2+2?")
print(result.response) # '4'
print(result.model) # 'moonshot/kimi-k2.6' (Moonshot flagship — vision + reasoning_content)
print(f"Saved {result.routing.savings * 100:.0f}%") # 'Saved 94%'
# Auto-routes to the cheapest capable model
result = client.smart_chat("Summarize this changelog entry in one line")
print(result.response)
print(result.model) # 'google/gemini-2.5-flash'
print(result.routing.task_type) # 'chat'
print(f"Saved {result.routing.savings * 100:.0f}%") # 'Saved 90%'

# Complex reasoning task -> routes to reasoning model
# Complex reasoning task -> routes to a reasoning model
result = client.smart_chat("Prove the Riemann hypothesis step by step")
print(result.model) # 'deepseek/deepseek-reasoner'
print(result.model) # 'deepseek/deepseek-v4-pro'
```

Want to see the decision without paying for a call? `client.route(...)` runs the
same routing locally and returns the decision only:

```python
decision = client.route("Prove the Riemann hypothesis step by step")
print(decision.model) # 'deepseek/deepseek-v4-pro'
print(decision.tier) # 'REASONING'
print(decision.task_type) # 'reasoning'
print(decision.candidates) # ordered chain; smart_chat walks it on a 5xx/timeout
print(decision.reasoning) # human-readable explanation of the pick
```

### Routing Profiles

| Profile | Description | Best For |
|---------|-------------|----------|
| `free` | NVIDIA free tier — smart-routes across <!-- br:models.free -->5<!-- /br:models.free --> models (DeepSeek V4 Pro/Flash, Nemotron Nano Omni, Qwen3, GLM-4.7, Llama 4, Mistral) | Zero-cost testing, dev, prod |
| `eco` | Cheapest models per tier (DeepSeek, NVIDIA) | Cost-sensitive production |
| `free` | NVIDIA free tier — smart-routes across the <!-- br:models.free -->5<!-- /br:models.free --> $0 models (Step 3.7 Flash, Mistral Nemotron, Nemotron Nano Omni / 9B / 12B VL) | Zero-cost testing, dev, prod |
| `eco` | Cheapest capable model per tier | Cost-sensitive production |
| `auto` | Best balance of cost/quality (default) | General use |
| `premium` | Top-tier models (OpenAI, Anthropic) | Quality-critical tasks |
| `premium` | Top-tier models (Anthropic, OpenAI, Moonshot) | Quality-critical tasks |

```python
# Use premium models for complex tasks
result = client.smart_chat(
"Write production-grade async Python code",
routing_profile="premium"
)
print(result.model) # 'openai/gpt-5.4'
print(result.model) # 'openai/gpt-5.3-codex'
```

### How It Works

ClawRouter uses a 14-dimension rule-based classifier to analyze each request:

- **Token count** - Short vs long prompts
- **Code presence** - Programming keywords
- **Reasoning markers** - "prove", "step by step", etc.
- **Technical terms** - Architecture, optimization, etc.
- **Creative markers** - Story, poem, brainstorm, etc.
- **Agentic patterns** - Multi-step, tool use indicators

The classifier runs in <1ms, 100% locally, and routes to one of four tiers:
Routing runs on [Router Core](https://github.com/BlockRunAI/router-core) — the
same product-neutral engine the TypeScript SDK and the BlockRun gateway use, so
an identical request routes identically across all three. It is 100% local and
takes <1ms; no extra model call is made to decide.

Three stages:

1. **Classify** — a <!-- br:clawrouter.dimensions -->15<!-- /br:clawrouter.dimensions -->-dimension weighted
scorer maps the request onto a capability tier (token count, code presence,
reasoning markers, technical terms, creative markers, agentic patterns, and
more), and a task classifier labels the *shape* of the work: `chat`,
`code_edit`, `code_agent`, `tool_agent`, `reasoning_math`, `long_context`,
`extraction`, `vision`, …
2. **Filter** — capability constraints are hard filters, not preferences. A
model that cannot hold the conversation, emit the requested output length,
call tools, or read images is dropped before scoring, so the router never
picks a model the request would fail on.
3. **Rank** — surviving candidates are scored on task affinity, cost, speed and
reliability. The winner serves the request; the rest become the ordered
fallback chain that `smart_chat` walks on a timeout or 5xx.

The four capability tiers:

| Tier | Example Tasks | Auto Profile Model |
|------|---------------|-------------------|
| SIMPLE | "What is 2+2?", definitions | moonshot/kimi-k2.6 |
| MEDIUM | Code snippets, explanations | google/gemini-2.5-flash |
| SIMPLE | Short questions, definitions | google/gemini-2.5-flash |
| MEDIUM | Code snippets, explanations | moonshot/kimi-k2.7 |
| COMPLEX | Architecture, long documents | google/gemini-3.1-pro |
| REASONING | Proofs, multi-step reasoning | deepseek/deepseek-reasoner |
| REASONING | Proofs, math, multi-step reasoning | deepseek/deepseek-v4-pro |

Every decision is explainable — `result.routing` carries the tier, the task
type, the confidence, the ranked `candidates`, the per-candidate
`candidate_scores` (quality / cost / speed / reliability) and a `reasoning`
string describing why that model won.

## How Payment Works

Expand Down Expand Up @@ -1671,8 +1701,8 @@ blockrun-llm is a Python SDK that provides pay-per-request access to 43+ large l
### How does payment work?
When you make an API call, the SDK automatically handles x402 payment. It signs a USDC transaction locally using your wallet private key (which never leaves your machine), and includes the payment proof in the request header. Settlement is non-custodial and instant on Base or Solana.

### What is smart routing / ClawRouter?
ClawRouter is a built-in smart routing engine that analyzes your request across <!-- br:clawrouter.dimensions -->15<!-- /br:clawrouter.dimensions --> dimensions and automatically picks the cheapest model capable of handling it. Routing happens locally in under 1ms. It can save up to <!-- br:savings.autoVsBaselinePct -->88<!-- /br:savings.autoVsBaselinePct -->% on LLM costs compared to using premium models for every request.
### What is smart routing / Router Core?
Router Core is BlockRun's built-in routing engine — shared with the TypeScript SDK and the gateway, so the same request routes the same way everywhere. It scores your request across <!-- br:clawrouter.dimensions -->15<!-- /br:clawrouter.dimensions --> dimensions, drops every model that can't actually handle it (context, output length, tools, vision), then picks the cheapest capable one and keeps the rest as a fallback chain. Routing happens locally in under 1ms and makes no extra model call. It can save up to <!-- br:savings.autoVsBaselinePct -->88<!-- /br:savings.autoVsBaselinePct -->% on LLM costs compared to using premium models for every request.

### How much does it cost?
Pay only for what you use. Prices start at **FREE** (11 NVIDIA-hosted models). Paid models start at $0.10/M tokens. There are no minimums, subscriptions, or monthly fees. $5 in USDC gets you thousands of requests.
Expand Down
2 changes: 1 addition & 1 deletion VERSION
Original file line number Diff line number Diff line change
@@ -1 +1 @@
1.10.1
1.11.0
6 changes: 5 additions & 1 deletion blockrun_llm/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -101,6 +101,8 @@
APIError,
AudioModel,
AudioTrack,
# Smart routing types
CandidateScore,
ChatChunkChoice,
ChatChunkDelta,
ChatChunkFunctionCall,
Expand Down Expand Up @@ -184,7 +186,7 @@
create_wallet as generate_wallet, # User-friendly alias
)

__version__ = "1.10.1"
__version__ = "1.11.0"
__all__ = [
"NETWORK_ALIASES",
"SUPPORTED_NETWORKS",
Expand All @@ -196,6 +198,8 @@
"AsyncSolanaLLMClient",
"AudioModel",
"AudioTrack",
# Smart routing types
"CandidateScore",
"ChatChunkChoice",
"ChatChunkDelta",
"ChatChunkFunctionCall",
Expand Down
79 changes: 58 additions & 21 deletions blockrun_llm/client.py
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@
from dotenv import load_dotenv
from eth_account import Account

from .router import route as route_request
from .router_adapter import BASE_MINIMUM_PAYMENT_USD, route_with_catalog
from .tx_log import (
TransactionLogger,
_resolve_log_dir,
Expand Down Expand Up @@ -490,6 +490,10 @@ def _get_model_pricing(self) -> dict[str, dict[str, float]]:
pricing: dict[str, dict[str, float]] = {}
for model in models:
model_id = model.get("id", "")
# A model the catalog marks unavailable must not win routing — every
# smart call to it would fail with a non-transient error.
if model.get("available") is False:
continue
block = model.get("pricing") or {}
input_price = block.get("input", model.get("inputPrice", model.get("input_price", 0)))
output_price = block.get(
Expand All @@ -504,6 +508,38 @@ def _get_model_pricing(self) -> dict[str, dict[str, float]]:
self._model_pricing_cache = pricing
return pricing

def route(
self,
prompt: str,
*,
system: str | None = None,
max_tokens: int | None = None,
routing_profile: RoutingProfile = "auto",
requires_structured_output: bool = False,
) -> RoutingDecision:
"""
Inspect a routing decision without making or paying for a model call.

The first invocation may fetch the public model catalog for current
prices; routing itself is local and costs nothing.

Example:
decision = client.route("Prove the Riemann hypothesis")
print(decision.model) # 'deepseek/deepseek-v4-pro'
print(decision.task_type) # 'reasoning'
print(decision.candidates) # ordered fallback chain
"""
decision = route_with_catalog(
prompt,
system,
max_tokens or self.DEFAULT_MAX_TOKENS,
self._get_model_pricing(),
routing_profile=routing_profile,
requires_structured_output=requires_structured_output,
minimum_payment_usd=BASE_MINIMUM_PAYMENT_USD,
)
return RoutingDecision(**decision)

def smart_chat(
self,
prompt: str,
Expand All @@ -516,27 +552,32 @@ def smart_chat(
"""
Smart chat with automatic model routing.

Routes requests to the cheapest capable model using ClawRouter's
14-dimension rule-based scoring algorithm (<1ms, 100% local).
Uses BlockRun's product-neutral Router Core portfolio strategy — the
same engine the TypeScript SDK and the gateway run. It classifies the
task shape locally (<1ms, no extra model call), enforces capability
constraints as hard filters, and ranks an ordered candidate portfolio:
the cheapest model that can handle the request wins, and the rest become
the transient-error fallback chain.

Args:
prompt: User message
system: Optional system prompt
max_tokens: Max tokens to generate (default: 1024)
temperature: Sampling temperature
routing_profile: "free" | "eco" | "auto" | "premium"
- free: nvidia/gpt-oss-120b only (FREE)
- eco: Cheapest models per tier (DeepSeek, xAI)
- free: NVIDIA's $0 models only — no wallet needed
- eco: Cheapest capable model per tier
- auto: Best balance of cost/quality (default)
- premium: Top-tier models (OpenAI, Anthropic)
- premium: Top-tier models (Anthropic, OpenAI, Moonshot)

Returns:
SmartChatResponse with response, model, and routing decision

Example:
result = client.smart_chat("What is 2+2?")
print(result.response) # '4'
print(result.model) # 'google/gemini-2.5-flash'
print(result.response) # '4'
print(result.model) # 'google/gemini-3.5-flash'
print(result.routing.method) # 'portfolio'
print(f"Saved {result.routing.savings * 100:.0f}%")

# With routing profile
Expand All @@ -545,22 +586,18 @@ def smart_chat(
routing_profile="premium" # Use top-tier models for complex tasks
)
"""
# Get model pricing for routing decision
model_pricing = self._get_model_pricing()
max_output_tokens = max_tokens or self.DEFAULT_MAX_TOKENS

# Route the request
decision = route_request(
prompt=prompt,
system_prompt=system,
max_output_tokens=max_output_tokens,
model_pricing=model_pricing,
decision = route_with_catalog(
prompt,
system,
max_tokens or self.DEFAULT_MAX_TOKENS,
self._get_model_pricing(),
routing_profile=routing_profile,
minimum_payment_usd=BASE_MINIMUM_PAYMENT_USD,
)

# Make the chat request with selected model. Pass the tier's remaining
# models as fallbacks so a hung upstream (e.g. NVIDIA NIM) doesn't
# hard-fail when smart_chat could just walk to the next visible model.
# Make the chat request with selected model. Pass the remaining ranked
# candidates as fallbacks so a hung upstream (e.g. NVIDIA NIM) doesn't
# hard-fail when smart_chat could just walk to the next capable model.
response = self.chat(
model=decision["model"],
prompt=prompt,
Expand Down
Loading
Loading