Skip to content

Add Modal support: run the full benchmark on cloud GPUs#59

Closed
arnavw wants to merge 2 commits into
aisa-group:mainfrom
arnavw:modal-adapter
Closed

Add Modal support: run the full benchmark on cloud GPUs#59
arnavw wants to merge 2 commits into
aisa-group:mainfrom
arnavw:modal-adapter

Conversation

@arnavw

@arnavw arnavw commented Jul 7, 2026

Copy link
Copy Markdown

Runs the full pipeline (agent → contamination judge → evaluation) on Modal rented H100s, for anyone without cluster access. Purely additive — everything lives in src/modal_adapter/ (same shape as the Harbor adapter in #8) and the condor path is untouched. Each phase script is a near line-for-line port of the corresponding section of run_task.sh, and results land in the exact EVAL_DIR layout, so scripts/collect.py works on downloaded results unchanged.

  • One run = a chain of three Modal Functions: run_agent (standard image, 1–8×H100, ≤22h), run_judge (fresh CPU container, so agent tampering can't leak into the judge), run_eval (vllm_debug image, retry ladder as-is). Submissions are fire-and-forget from a laptop.
  • Both .def files are translated to modal.Image chains; the HF cache is a Modal Volume seeded by running download_resources.py inside Modal, mounted read-only and exposed through a symlink farm (fuse-overlayfs can't mount in Modal's gVisor runtime — verified empirically).
  • ~$55–60 per standard 10h run at list prices; full setup, usage, and a table of deviations from the cluster (preemption semantics, judge on CPU, run-id format, 22h cap) are in src/modal_adapter/README.md.

Verified so far: both images build and deploy; CPU smoke checks pass on both images (imports, pinned CLI versions, volumes, secret wiring, symlink farm); the repo's check_cuda.py/check_cuda_writing.py pass on a real H100. Not yet run: a paid end-to-end run — the README's smoke sequence (seed minimal cache, 1h claude run on gsm8k, ~$10–15) is the intended validation before this leaves draft.

Two notes for reviewers: standard.def installs inspect_evals from HEAD, which now requires Python ≥3.11 and would break a fresh cluster build too — the Modal image pins the same commit vllm_debug.def pins. And --torch-backend=auto needs a CUDA driver at build time, so the Modal images resolve vLLM from PyPI instead.

🤖 Generated with Claude Code

Arnav Wadehra and others added 2 commits July 7, 2026 15:48
Ports run_task.sh to a chain of three Modal Functions (agent -> judge ->
eval) with the two container images translated from the .def files.
fuse-overlayfs is replaced by a symlink farm over a read-only HF cache
volume (FUSE mounts are unavailable in Modal's gVisor runtime).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@arnavw arnavw closed this Jul 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant