Add Modal support: run the full benchmark on cloud GPUs#59
Closed
arnavw wants to merge 2 commits into
Closed
Conversation
Ports run_task.sh to a chain of three Modal Functions (agent -> judge -> eval) with the two container images translated from the .def files. fuse-overlayfs is replaced by a symlink farm over a read-only HF cache volume (FUSE mounts are unavailable in Modal's gVisor runtime). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Runs the full pipeline (agent → contamination judge → evaluation) on Modal rented H100s, for anyone without cluster access. Purely additive — everything lives in
src/modal_adapter/(same shape as the Harbor adapter in #8) and the condor path is untouched. Each phase script is a near line-for-line port of the corresponding section ofrun_task.sh, and results land in the exactEVAL_DIRlayout, soscripts/collect.pyworks on downloaded results unchanged.run_agent(standard image, 1–8×H100, ≤22h),run_judge(fresh CPU container, so agent tampering can't leak into the judge),run_eval(vllm_debug image, retry ladder as-is). Submissions are fire-and-forget from a laptop..deffiles are translated tomodal.Imagechains; the HF cache is a Modal Volume seeded by runningdownload_resources.pyinside Modal, mounted read-only and exposed through a symlink farm (fuse-overlayfs can't mount in Modal's gVisor runtime — verified empirically).src/modal_adapter/README.md.Verified so far: both images build and deploy; CPU smoke checks pass on both images (imports, pinned CLI versions, volumes, secret wiring, symlink farm); the repo's
check_cuda.py/check_cuda_writing.pypass on a real H100. Not yet run: a paid end-to-end run — the README's smoke sequence (seed minimal cache, 1h claude run on gsm8k, ~$10–15) is the intended validation before this leaves draft.Two notes for reviewers:
standard.definstalls inspect_evals from HEAD, which now requires Python ≥3.11 and would break a fresh cluster build too — the Modal image pins the same commitvllm_debug.defpins. And--torch-backend=autoneeds a CUDA driver at build time, so the Modal images resolve vLLM from PyPI instead.🤖 Generated with Claude Code