The generator fix in #145 addresses #144 for new exports. Existing published tasks need a separate migration; changing the library cannot remove files baked into an old task image.
The current manifest for AdithyaSK/repo2rlenv-pr-diff lists 181 tasks. On 2026-09-22, the sampled ripgrep task Dockerfile at revision 971b463606b0ffa69ac6b6125d90c6868744be2b still embeds /verifier/oracle.patch in the agent image. This confirms the published dataset has not been migrated; it is not a claim that all 181 tasks were individually inspected.
Required follow-up:
- Audit all published PR-diff tasks and any copies for baked grading assets.
- Move oracle/verifier assets into
tests/, update the evaluation script, and clear both reward formats before grading.
- Regenerate hashes/manifests where applicable, rebuild affected images, and invalidate old image caches.
- Check no-op and oracle behavior, and confirm the oracle is absent during the agent phase.
- Publish a new dataset revision with migration notes. Mark old revisions as affected; previous oracle passes do not establish resistance to answer leakage.
Keep this open until the repaired dataset revision and validation evidence are linked here.
The generator fix in #145 addresses #144 for new exports. Existing published tasks need a separate migration; changing the library cannot remove files baked into an old task image.
The current manifest for AdithyaSK/repo2rlenv-pr-diff lists 181 tasks. On 2026-09-22, the sampled ripgrep task Dockerfile at revision 971b463606b0ffa69ac6b6125d90c6868744be2b still embeds
/verifier/oracle.patchin the agent image. This confirms the published dataset has not been migrated; it is not a claim that all 181 tasks were individually inspected.Required follow-up:
tests/, update the evaluation script, and clear both reward formats before grading.Keep this open until the repaired dataset revision and validation evidence are linked here.