Skip to content

migrate: repair pr_diff tasks baked before the #145 oracle-isolation fix - #157

Open
KNambiarDJsc wants to merge 1 commit into
huggingface:mainfrom
KNambiarDJsc:fix/migrate-pr-diff-oracle-leak
Open

KNambiarDJsc wants to merge 1 commit into
huggingface:mainfrom
KNambiarDJsc:fix/migrate-pr-diff-oracle-leak

Conversation

@KNambiarDJsc

Copy link
Copy Markdown
Contributor

Summary

Closes the tooling gap #145 explicitly left open: "Existing published datasets... need re-emission to pick up the fix. A migration/re-bake tool could be a follow-up." — this is that follow-up, for #155.

repo2rlenv migrate pr-diff <dir> [--apply] walks a dataset directory, detects the pre-#145 baked-oracle Dockerfile (base64 -d > /verifier/oracle.patch), and repairs it:

  • Rewrites environment/Dockerfile, tests/test.sh, and adds tests/{verifier.py,oracle.patch,instruction.md} using the exact same builder functions generate uses today (build_pr_diff_environment_dockerfile, build_pr_diff_eval_script, _pr_diff_aux_files) — a migrated task is byte-identical to one freshly emitted, not a hand-maintained parallel implementation that could drift from the real fix.
  • instruction.md and solution/patch.diff (the oracle itself) are never touched, so content_hash (which only ever covers those two files — emitter/harbor.py:_content_hash) is unchanged; a migrated task keeps its original identity.
  • repo_url/base_commit are recovered from the existing Dockerfile's remote set-url origin line and task.toml's metadata.repo2env.ref — both untouched by the leak fix, so nothing is guessed.
  • Refuses to touch a task whose instruction.md/solution/patch.diff don't hash to the content_hash its task.toml already claims, rather than silently rewriting something inconsistent — surfaced as action: "error".
  • Defaults to a dry-run audit report; --apply writes. Idempotent — a task already on the fixed shape reports already_safe and is untouched.

Scope note

Per #155's checklist, this PR covers detection + file repair only:

  • Not covered here (needs the maintainer's own credentials/infra): rebuilding affected Docker images, invalidating registry caches, and publishing the repaired revision to AdithyaSK/repo2rlenv-pr-diff. Those are exactly the steps this tool's output is meant to feed into.
  • Not verified here: "confirm the oracle is absent during the agent phase" for a real migrated task's built image — that needs Docker, which isn't available in the environment I built/tested this in. The rewritten Dockerfile is byte-for-byte what generate produces today, whose end-to-end absence-of-oracle was already verified in pr_diff: ship oracle + verifier via tests/, not baked in image #145's own test plan, but I haven't independently re-verified it against a migrated (as opposed to freshly generated) task.
  • "Any copies": I checked the repo's own docs/RFCs/README for other referenced pr_diff datasets and found only AdithyaSK/repo2rlenv-pr-diff (181 tasks). If there are other copies/forks on the Hub, I don't have visibility into that from the repo alone.

Test plan

  • uv run --all-extras pytest tests/test_pipeline_pr_diff.py -q — 36 passed (9 new: detects+repairs, preserves content_hash, idempotent re-run, dry-run writes nothing, skips non-pr_diff tasks, already-safe task is a no-op, flags content_hash mismatch instead of rewriting, missing-task.toml error)
  • uv run --all-extras pytest -q (full suite) — 2003 passed, 0 failed, 6 skipped (all live-network/Docker-gated)
  • ruff check . / ruff format --check . — clean
  • Manual end-to-end smoke test: built a synthetic pre-pr_diff: ship oracle + verifier via tests/, not baked in image #145 task fixture, ran migrate pr-diff dry-run → apply → re-apply (idempotent) → validate --deep on the result, all as expected

…-isolation fix

huggingface#145 stopped baking the oracle/verifier/instruction into pr_diff's agent
image, but already-published tasks (e.g. AdithyaSK/repo2rlenv-pr-diff, 181
tasks) keep the pre-fix Dockerfile until re-emitted — huggingface#155.

repo2rlenv migrate pr-diff <dir> [--apply] detects the baked-oracle marker
and rewrites environment/Dockerfile + tests/{test.sh,verifier.py,
oracle.patch,instruction.md} via the SAME builder functions fresh generate
calls today, so a migrated task is byte-identical to one emitted now, not
a hand-maintained parallel implementation. instruction.md and
solution/patch.diff (the oracle itself) are never touched, so a migrated
task keeps its original content_hash. repo_url/base_commit are recovered
from the existing Dockerfile's remote set-url line and task.toml's
metadata.repo2env.ref, not re-derived or guessed. Refuses to touch a task
whose instruction.md/solution/patch.diff don't hash to the content_hash
task.toml already claims, rather than silently rewriting an inconsistent
bundle. Defaults to a dry-run report; --apply writes.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant