LC_SRUN_OVERLAP unset in recipe environment: recipes srun-deadlock behind the dask worker step inside an allocation

[cailmdaley]

What happened

Recipes written per the documented HPC pattern — srun $LC_SRUN_OVERLAP -A … <cmd> — deadlock when materialized through lc run inside a SLURM allocation. lc run srun-launches its dask worker with --nthreads=<all allocation CPUs> (a step holding the full allocation), and nothing sets LC_SRUN_OVERLAP in the worker's environment, so the recipe's inner srun expands to a plain step request that waits forever for CPUs the worker step never releases. Observed as a 30-minute silent hang (recipe step pending, squeue -s showing the dask step holding all cores); it would have waited until the allocation's walltime.

Workaround that fixed it: manually export LC_SRUN_OVERLAP="--overlap" in the environment that invokes lc run (it propagates through srun → worker → recipe shell).

Reproduction

1. salloc -N1 -c<N> --no-shell; srun --jobid=$JOBID --overlap lc run <output> where the output's recipe launches srun $LC_SRUN_OVERLAP … <heavy step>.
2. Recipe's step queues indefinitely behind the dask worker step; no error, no progress.

Suggestion

lc run should set LC_SRUN_OVERLAP=--overlap itself (or launch its dask workers with --overlap) whenever it detects SLURM_JOB_ID and srun-launches workers — the variable is part of the documented recipe contract, so the engine that spawns the recipe environment should guarantee it.

Environment

• lightcone-cli: 0.3.8.dev25+ge403f216d
• Python: 3.14 (uv tool env)
• OS: Linux 4.18.0 (Leonardo, CINECA / SLURM)

— Claude (Fable) on behalf of Cail