feat: Parsl-based simulation orchestration (Issue #16)

[gregorweiss]

Closes #16. Replaces Nextflow with Python-native Parsl workflow for GROMACS campaigns on SLURM.

Core: EM → NVT → NPT → Production chained via inputs=[prev_future]; independent systems run in parallel. Checkpoint detection skips completed stages before opening a Parsl session.

Notable decisions:

• stage_overrides in SlurmExecutorConfig for per-stage CPU/walltime — the same knob Issue Implement scalability benchmark sweep #9 benchmark sweep will tune
• Sequential-only session constraint (parsl.clear() on exit) — enables benchmark → production pattern without process restart
• find_structure_file promoted to public API in mdfactory.orchestration — prep for performance/benchmark.py

Partial-campaign builds: systems with missing build files are skipped with a warning rather than aborting the run.

64 unit tests; integration-tested on cluster (mixedbox tutorial campaign).

[gregorweiss]

Progress Update

Refactored the TUI SLURM wizard's stage-override prompts to be context-aware:

• Replaced the boolean show_stage_overrides with a stages: tuple[str, ...] | None parameter that flows through the entire call chain (_resolve_slurm_flag → configure_slurm_interactive → _configure_with_cluster/_configure_manual → _prompt_stage_overrides)
• Added _resolve_stages() helper that resolves None to the full pipeline from STAGE_REGISTRY
• _prompt_stage_overrides now returns {} immediately if len(stages) < 2 — no question asked
• EM/GPU hint panel only shown when both GPU is selected AND EM is in the active stages
• build passes stages=() — no stage prompts at all
• simulate passes the user's --stages selection (or None for full pipeline)
• Single-stage runs (e.g. --stages Production) skip overrides automatically
• Future workflows get correct behavior without call-site changes

Commit: 836d3dd

---

Progress tracked by mach6

[gregorweiss]

Progress Update

Added rescue retry integration tests that validate the mechanism with real GROMACS execution.

Test suite (test_rescue_integration.py)

5 tests in TestRescueIntegrationEM:

1. test_grompp_succeeds — sanity check that fixture files are valid
2. test_em_crashes_at_tier_0 — confirms oversized emstep=0.5 causes a physics failure
3. test_apply_rescue_tier_produces_valid_mdp — verifies MDP parameter modification (halved emstep, doubled nsteps)
4. test_em_succeeds_with_rescue_tier — iterates tiers 1-4 until EM converges, proving the rescue strategy works
5. test_classify_failure_detects_physics — validates error classification recognises the crash as PHYSICS

Design decisions

• Dynamic force field detection: topology is generated at test time by probing the local GROMACS installation (tries oplsaa, amber99sb-ildn, gromos54a7, charmm27)
• Marked slow: excluded from default test suite, run with pixi run pytest -m slow
• Skips gracefully: skipif(gmx not on PATH) + force field detection skip
• No Parsl dependency: tests validate rescue MDP logic + real GROMACS execution directly via subprocess (Phase 1 approach from the plan)

Fixture: 3 SPC waters in 1nm box

• system.pdb — valid .gro format, 3 waters with ~0.3nm separation
• em.mdp — emstep=0.5 (50x normal), guaranteed to crash
• topology.top — placeholder, overwritten by fixture with detected FF

Commit: d4a116e

---

Progress tracked by mach6

[gregorweiss]

Progress Update

Added interactive GROMACS detection to the SLURM TUI wizard (simulate only, skipped for build):

• _detect_gromacs_modules() — discovers available modules via module avail gromacs
• _prompt_gromacs_source() — warns when gmx/gmx_mpi not on PATH, lets user select a module or enter a custom command
• Selected module is prepended to the worker_init default (e.g. module load gromacs/2024.4; eval "$(pixi shell-hook ...)")
• Gated on stages != () — only prompts during simulate, never during build (which is pure Python and doesn't need GROMACS)
• Replaced the silent $LOADEDMODULES-based for_simulate approach with an explicit interactive prompt — catches the case where the user hasn't loaded GROMACS on the login node yet

Commit: 94c1632

---

Progress tracked by mach6

[gregorweiss]

Progress Update

Issues closed

Audited acceptance criteria for issues 30, 32, and 34 — all criteria met:

• Issue 30 (adaptive rescue retry): 9/9 criteria met. Added missing --max-rescue CLI passthrough test (ee972c1), ticked all boxes, closed.
• Issue 32 (per-stage override prompts): 9/9 criteria met. Ticked all boxes, closed.
• Issue 34 (DependencyError unwrapping): 4/4 criteria met. Ticked all boxes, closed.

Formatting & linting

Ran pre-commit (ruff lint + ruff-format) across the branch — fixed 4 lint violations and reformatted 8 files. All 239 orchestration tests pass.

Commits: ee972c1 (test), 1b7e884 (style)

---

Progress tracked by mach6

[gregorweiss]

Progress Update

Closed the remaining acceptance criteria gaps from Issue #16:

Dry-run with resolved commands

_log_dry_run_plan now prints the actual gmx grompp / gmx mdrun command lines that would be submitted for each stage (per simulation), not just the plan summary. Uses the pure script builders (_build_grompp_script, _build_mdrun_script) to resolve all flags — GPU mode, thread count, restart parameters, reference structures — and extracts the $GMX_BIN invocation for display.

Findings write-up

Added docs/content/docs/user-guide/parsl-findings.mdx documenting:

• What works (build orchestration, simulation orchestration, infrastructure)
• What doesn't / has limitations (per-stage SLURM allocations, custom node packing, monitoring)
• Gaps vs. requirements matrix
• Architecture decisions
• Test coverage summary

Custom workflow composition

Deferred to Issue #10 (GPU MPS) — see comment on #16. The Parsl-native max_workers_per_node achieves basic full-node utilisation; explicit bash-level packing fits the MPS scenario.

Commits: 378c28d, aa37bb6

---

Progress tracked by mach6

[gregorweiss]

DESIGN.md Review: Simulation Orchestration Refactoring

Reviewed the full PR against DESIGN.md principles (lean, YAGNI, DRY, SOLID, readable, minimal dependencies). Applied 10 refactoring steps — net result: -554 lines, same functionality, all 257 tests pass.

Changes Made

#	Change	Impact
1	Removed dead code (_validate_simulation_dir, _default_walltime)	-25 lines
2	Eliminated MDP parsing duplication — _extract_expected_frames_from_mdp now uses mdp.py	-25 lines, DRY
3	Moved _describe_failure to errors.py (fixes dependency inversion)	SOLID
4	Moved _get_block_status to progress.py (fixes dependency inversion)	SOLID
5	Added _StageState TypedDict for checkpoint detection return values	Type safety
6	Extracted get_stage_config_or_none helper to config.py (DRY 3× pattern)	-6 lines, DRY
7	Removed convenience wrappers from stages.py (never used in production)	-155 lines, YAGNI
8	Trimmed verbose docstrings on private helpers to 1-line summaries	-250 lines, readable
9	Switched environment.py to loguru (consistency with all other modules)	Consistency
10	Extracted _has_restart_pair helper (DRY 4× pattern)	DRY

What Was NOT Changed (intentionally)

• run_simulations not split — it's long but linear with one caller
• No Protocol for get_stage_config — the helper suffices
• _build_mdrun_script/_build_grompp_script params not refactored — pure flag resolvers
• _detect_needed_stages kept — 6-line wrapper used by 12 tests

Verification

All orchestration tests pass (257 passed) after each step. No public API changes.

[maxscheurer]

LGTM 🚀