This repository was generated and its experiments were run with the OpenResearch CLI (
orx).
What is in
main: the exact code snapshot fromorx/faithful-epd-1x, which achieved the headline EPD result: 25.0% pass rate and 44.4% of the measured fresh-ICL gain captured. The original experiment branch remains preserved as its immutable provenance record.
This is a small, budget-aware replication of Sample-Efficient Learning from Agent Experience (Experience-to-Policy Distillation, EPD). It tests whether an agent can turn improvement from prior coding attempts—normally available only as long prompt context—into a LoRA adapter that works without that experience at evaluation time.
The faithful round uses Qwen/Qwen3.5-9B, Tinker LoRA (rank 16), 12 curated SWE-smith oauthlib tasks, official markers, and three fresh evaluation rollouts per task. Results are therefore 36 attempts per method and should be read as directional evidence rather than precise estimates.
For a recorded task trajectory tau_exp, the frozen teacher sees the accumulated
experience while the student does not. At each logged state h_t, the teacher
samples a recommended action a'_t; the student is trained to maximize the
probability of that action given only h_t:
loss = -sum_t log student(a'_t | h_t)
Here a'_t is the teacher's experience-informed recommendation at a real
logged state h_t. In branch packing, the recommendation is supervised,
then the original action and its real observation are appended only as masked
context for the next anchor. Thus packed data is an efficient annotated
training construction, not a coherent counterfactual environment rollout.
The original paper's main qualitative result does replicate at 1 sequence per trial: EPD captures a substantial part of the experience-ICL gain, while ordinary SFT on the logged trajectories does not. The paper's teacher-data scaling result does not reproduce in this small setting.
| Method | Pass rate | ICL-normalized gain (G_ICL) |
|---|---|---|
| Zeroshot | 5/36 = 13.9% | 0% |
| Fresh ICL | 14/36 = 38.9% | 100% |
| SFT, all trials | 5/36 = 13.9% | 0% |
| SFT, final trial only | 3/36 = 8.3% | -22% |
| EPD, 1 sequence/trial | 9/36 = 25.0% | 44.4% |
| EPD, 4 sequences/trial; proportional optimization | 1/36 = 2.8% | -44.4% |
| EPD, 4 sequences/trial; constant optimization | 6/36 = 16.7% | 11.1% |
Fresh ICL was the essential gate: it beat zeroshot by 25 percentage points. A residual premature-completion behavior makes that ICL number conservative: every rollout that engaged with the task passed, but many still stopped on the first turn.
The plot below mirrors the paper's teacher-data-scaling figure, but changes the question to the one this replication can answer: what happens when the same recorded experience is used to sample one versus four packed teacher sequences? The 4x variants share the same source trials; only their sampled teacher recommendations differ.
The proportional 4x run used four times as many sampled sequences and four times as many optimizer steps; it collapsed despite a healthy training loss. Holding 4x to the 1x update budget recovered much of the loss, but still did not beat 1x. The most plausible reading is overtraining on highly correlated teacher samples, with little marginal information from additional samples.
expdist/
agent.py agent prompt, tool use, and rollout loop
corpus.py retained persistent-worktree experience corpus
records.py trajectories, thinking, actions, and observations
training.py masked EPD/SFT datum construction and LoRA updates
train_eval.py train-then-evaluate experiment modes
benchmarks.py pinned SWE-smith task and official-marker integration
docker_env.py isolated coding-environment lifecycle
tinker_api.py Tinker sampling and retry/backoff handling
run.py fixed OpenResearch entry point
experiment.toml committed experiment mode and hyperparameters
tests/ focused training-data checks
docs/images/ README figures generated from logged run results
Every OpenResearch experiment uses the same committed run command:
uv run python -m expdist.runBehavior changes through committed experiment.toml and code, never by
changing the run command. TINKER_API_KEY is read only from the runtime
environment and is not committed.
Experiment branches are intentionally preserved: each completed branch records the exact code that produced its result. Start with the collection node, then the references, then compare the method siblings.
| Stage | Branch to visit | What it establishes |
|---|---|---|
| Persistent corpus | orx/persistent-worktree-collection |
12 retained tasks with evolving repositories and prior-trial history preserved |
| Zeroshot reference | orx/fresh-zeroshot-reference |
13.9% no-experience baseline |
| Fresh-ICL gate | orx/paper-protocol-fresh-icl |
38.9% experience-in-context reference |
| SFT control | orx/paper-protocol-sft |
Logged-trajectory SFT does not internalize the ICL gain |
| SFT final-only control | orx/paper-protocol-sft-final-only |
Tests whether only polished successful trials suffice |
| EPD 1x | orx/faithful-epd-1x |
Best configuration: 25.0%, retaining 44.4% of the ICL gain |
| EPD 4x | orx/faithful-epd-4x |
Four teacher sequences per trial with proportional updates: collapse |
| 4x optimization control | orx/epd-4x-constant-optimization |
Same 4x data with the 1x update budget: partial recovery |
The detailed report is available in the OpenResearch dashboard's Files tab at
project/faithful-replication/report.md.
Earlier rounds were not treated as evidence because they violated key details of the paper's protocol. The reported round fixes the following:
- capture and train on Qwen3.5 reasoning tokens rather than silently replacing them with empty thinking blocks;
- retain one repository and interaction history across a task's trials;
- tell fresh-ICL rollouts explicitly that the repository has been reset;
- use sampled Enhanced-Teacher-Reasoning targets and paper-style branch packing;
- assert that only teacher decision tokens—not the prompt or recorded trajectory—receive loss.
Use OpenResearch to create a child experiment, commit the changed
experiment.toml or implementation, and run the fixed command through orx.
Do not edit or merge a completed experiment branch: those branches are frozen
provenance records. New ideas should descend from the relevant result branch.
The most useful next experiments are a second evaluation seed for EPD-1x and teacher-target deduplication or diversification before attempting further sequence scaling.