Skip to content

Repair published PR-diff tasks after the oracle isolation fix #155

Description

@adithya-s-k

The generator fix in #145 addresses #144 for new exports. Existing published tasks need a separate migration; changing the library cannot remove files baked into an old task image.

The current manifest for AdithyaSK/repo2rlenv-pr-diff lists 181 tasks. On 2026-09-22, the sampled ripgrep task Dockerfile at revision 971b463606b0ffa69ac6b6125d90c6868744be2b still embeds /verifier/oracle.patch in the agent image. This confirms the published dataset has not been migrated; it is not a claim that all 181 tasks were individually inspected.

Required follow-up:

  • Audit all published PR-diff tasks and any copies for baked grading assets.
  • Move oracle/verifier assets into tests/, update the evaluation script, and clear both reward formats before grading.
  • Regenerate hashes/manifests where applicable, rebuild affected images, and invalidate old image caches.
  • Check no-op and oracle behavior, and confirm the oracle is absent during the agent phase.
  • Publish a new dataset revision with migration notes. Mark old revisions as affected; previous oracle passes do not establish resistance to answer leakage.

Keep this open until the repaired dataset revision and validation evidence are linked here.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions