feat: ECHO env-token world modeling (RFC 010 + runnable demo)

[thegovind]

Draft on my fork to record a walkthrough before opening upstream. Proposal: #14.

Walkthrough

Video placeholder. Drag the recording here before upstream submission.

What this PR adds to OpenEnv

1. RFC 010 (rfcs/010-echo-env-token-world-model.md): a small, additive amendment so a trajectory can carry per-token role masks (action / env_output / warning) and the optimizer seam can take a world_model_coeff. This is the one thing OpenEnv is missing to train on environment tokens.
2. examples/echo_world_model/: a CPU reference that trains a small model on the environment's own tokens (verifier-free), with adapter notes for SkyRL, Tinker, and Foundry Fine-Tuning.
3. examples/echo_on_agent_world_model/: the same masks on the real upstream agent_world_model_env (10 CPU tests), showing its observations already carry the roles.

Purely additive. world_model_coeff = 0 drops the env term entirely, so the trainer's existing action-token objective is unchanged.

ECHO, crisply

During agent RL we mask out the environment's reply tokens and train only on the agent's actions. ECHO keeps that half: a small cross-entropy loss that makes the policy predict the environment's observation tokens. Those logits are already computed when the policy conditions on them, so the signal needs no extra rollouts and no teacher (about free).

flowchart LR
  R["one rollout<br/>shared logits"] --> A["action tokens"]
  R --> E["env observation tokens"]
  A --> LG["L_GRPO<br/>sparse, reward-gated"]
  E --> LE["lambda x CE<br/>dense, about free"]
  LG --> S["one optim step"]
  LE --> S
  S --> U["grad(L_GRPO) + grad(lambda x CE_env)"]

Loading

L = L_GRPO(action tokens) + lambda * CE(observation tokens)

The gap it closes. OpenEnv trajectories are message-level today, so a trainer cannot tell, per token, which tokens are actions and which are observations. RFC 010 Part A adds the masks (extends RFC 009); Part B adds the world_model_coeff knob on the optimizer seam (extends RFC 007).

Evidence

• Real env: 89% of the learnable tokens are environment observations that standard RL discards (7.9x the action tokens; 92% with a BPE tokenizer).
• It learns, verifier-free: held-out env-token CE 6.18 -> 0.27 nats/token on a small model.
• In one private run (Qwen3.5-4B, Tinker-style forward_backward + optim_step on Foundry Fine-Tuning, not reproducible from this PR): plain GRPO degraded held-out env-CE (10.08 -> 21.16, +110%) while ECHO learned it (10.08 -> 7.69, -24%). Config, seed, and logs available on request.
• Invariant tests: 6 (terminal) + 10 (AWM) pass on CPU (mask disjointness, lambda scaling, verifier-free, differentiability). They pin the contract, not the headline numbers.

Production rollouts can be isolated in ACA Sandboxes (PR #4, not part of this PR); the CPU demos here run in-process. Training adapters: SkyRL (open reference), Tinker, Foundry Fine-Tuning. Trained task-reward numbers (about 2.3x faster RL, TerminalBench-2.0 pass@1 about doubles) are in microsoft/echo-rl and arXiv:2605.24517.

Verify

# real-env demo
cd examples/echo_on_agent_world_model
python -m venv .venv && . .venv/bin/activate && pip install -r requirements.txt
python run_demo.py && pytest -q            # 10 passed

# toy reference that trains
cd ../echo_world_model
python -m venv .venv && . .venv/bin/activate && pip install -r requirements.txt
pytest -q && python train_echo.py --steps 60 --seed 0   # 6 passed

Credits

The real-env demo builds on the upstream Agent World Model env (agent_world_model_env) from Snowflake AI Research and UNC-Chapel Hill: AgentWorldModel-1K, CC-BY-4.0, arXiv:2602.10090. The ECHO objective is from microsoft/echo-rl (arXiv:2605.24517). Full citations in the example READMEs.

DCO signed. Refs #14, #9, #11. On the fork, this supersedes #17 (folded in here).

[thegovind]

📖 Added a diagram-rich explainer for the blog: examples/echo_world_model/EXPLAINER.md — 13 (validated) mermaid diagrams covering the idea, the loss, the OpenEnv per-token-role-mask gap, the demo wiring, the verifier-free result, and the full hybrid loop.

Harbor → ACA Sandboxes: Harbor is just SkyRL’s terminal sandbox backend (spin up a container, run the command, return the observation, run the verifier). So we map it to the ACASandboxProvider from #4 / huggingface#793 — the Azure-native, governed, default-deny option. See backends/aca-sandboxes.md. This is how the ECHO blog showcases the cloud-sandbox PR alongside it.

Scenario: reframed as forward-deployed incident triage (an FDE-relatable, ECHO-native terminal setting).

[thegovind]

🧵 End-to-end showcase added: E2E_SHOWCASE.md — one hill-climbing loop (incident-response agent) threading OpenEnv + ACA Sandboxes (#4) + ECHO + Loom/Tinker + Foundry optimization with a small owned model, plus a runnable-today-vs-GPU/Azure matrix.

🔧 Loom/ECHO is feasible with the current SDK. The azure-ai-finetuning-sessions SDK already exposes LossFn.CROSS_ENTROPY + LossFn.IMPORTANCE_SAMPLING, per-token weights/advantages, and gradient accumulation across forward_backward calls. So ECHO = two accumulated passes per step (GRPO importance_sampling on action tokens + λ-scaled cross_entropy on observation tokens — the cookbook already builds per-token weights, today zeroed on obs tokens). Recipe sketch is in the e2e doc.

[thegovind]

Superseded by the upstream submission: huggingface#819. Squashed the history into one clean DCO-signed commit, swapped the video placeholder for the reproducible env-token CE chart, and fixed the cross-repo references. Closing this fork staging PR; huggingface#819 is canonical.