feat: ECHO env-token world modeling (RFC 010 + two runnable examples)

[thegovind]

What this PR adds to OpenEnv

1. RFC 010 (rfcs/010-echo-env-token-world-model.md): a small, additive amendment so a trajectory can carry per-token role masks (action / env_output / warning) and the optimizer seam can take a world_model_coeff. This is the one thing OpenEnv is missing to train on environment tokens.
2. examples/echo_world_model/: a CPU reference that trains a small model on the environment's own tokens (verifier-free), with adapter notes for SkyRL, Tinker, and Foundry Fine-Tuning.
3. examples/echo_on_agent_world_model/: the same masks on the Agent World Model env, showing its observations already carry the roles.

Purely additive. world_model_coeff = 0 drops the env term entirely, so the trainer's existing action-token objective is unchanged.

ECHO, crisply

During agent RL we mask out the environment's reply tokens and train only on the agent's actions. ECHO keeps that half: a small cross-entropy loss that makes the policy predict the environment's observation tokens. Those logits are already computed when the policy conditions on them, so the signal needs no extra rollouts and no teacher (about free).

flowchart LR
  R["one rollout<br/>shared logits"] --> A["action tokens"]
  R --> E["env observation tokens"]
  A --> LG["L_GRPO<br/>sparse, reward-gated"]
  E --> LE["lambda x CE<br/>dense, about free"]
  LG --> S["one optim step"]
  LE --> S
  S --> U["grad(L_GRPO) + grad(lambda x CE_env)"]

Loading

L = L_GRPO(action tokens) + lambda * CE(observation tokens)

The gap it closes. OpenEnv trajectories are message-level today, so a trainer cannot tell, per token, which tokens are actions and which are observations. RFC 010 Part A adds the masks (extends RFC 009); Part B adds the world_model_coeff knob on the optimizer seam (extends RFC 007).

Results

The CPU reference trains a small model to predict the environment's own observation tokens, verifier-free (no grader, no teacher). Held-out env-token cross-entropy drops from 6.18 to 0.27 nats/token in about 40s on a laptop, reproducible with python train_echo.py --steps 60 --seed 0:

Beyond the toy:

• Real env: on the Agent World Model env, 89% of the learnable tokens are environment observations that standard agent-RL discards (7.9x the action tokens; 92% with a BPE tokenizer).
• One private fine-tuning run (Qwen3.5-4B, Tinker-style forward_backward + optim_step on Foundry Fine-Tuning; not reproducible from this PR): plain GRPO degraded held-out env-CE (10.08 -> 21.16, +110%) while ECHO learned it (10.08 -> 7.69, -24%). Config, seed, and logs available on request.
• Invariant tests: 6 (terminal) + 10 (AWM) pass on CPU (mask disjointness, lambda scaling, verifier-free, differentiability). They pin the contract, not the headline numbers.

Production rollouts can be isolated in ACA Sandboxes (#793, not part of this PR); the CPU demos here run in-process. Training adapters: SkyRL (open reference), Tinker, Foundry Fine-Tuning. Trained task-reward numbers (about 2.3x faster RL, TerminalBench-2.0 pass@1 about doubles) are in microsoft/echo-rl and arXiv:2605.24517.

Verify

# real-env demo
cd examples/echo_on_agent_world_model
python -m venv .venv && . .venv/bin/activate && pip install -r requirements.txt
python run_demo.py && pytest -q            # 10 passed

# toy reference that trains
cd ../echo_world_model
python -m venv .venv && . .venv/bin/activate && pip install -r requirements.txt
pytest -q && python train_echo.py --steps 60 --seed 0   # 6 passed

Credits

The real-env demo builds on the upstream Agent World Model env (agent_world_model_env, #428) from Snowflake AI Research and UNC-Chapel Hill: AgentWorldModel-1K, CC-BY-4.0, arXiv:2602.10090. The ECHO objective is from microsoft/echo-rl (arXiv:2605.24517). Full citations in the example READMEs.

DCO signed.

cc @burtenshaw

[burtenshaw]

Really nice example. Looking forward to a blog post or video on this.

[bot-ci-comment]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.