the whitelist lands under the sample that reads it, so no planned job owns a path a second instance deletes - #492
Merged
Conversation
… owns a path a second instance deletes The config records the registry name the KB chose and the consuming module builds the path, which is what lets that path be one no two jobs share: one owner for the location, so config and rule cannot disagree. `rule onlist` keeps temp(), keeps its verb, keeps no container, and gains a sample scope in both modules. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…s where the copy lands and when it goes RULES_THAT_MAY_OWN_A_SHARED_PATH is down to the load flag, the one path measured safe to race. The two claims a module's source text used to carry -- temp() and the producing verb -- move onto the rendered plan, where a rename cannot break them falsely and an indirection cannot pass them falsely, and the split-pool dry run gains the three paths in CB-position order as one argv token. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #488.
The barcode whitelist was the last rule output a whole deposit shared. Snakemake removes an output
before running the job that produces it, and reclaims a
temp()output when its last consumer inthat instance finishes — so a second instance over one results directory raced the first's
removal, not its write. That is the hazard class #478 measured on the index rule: 0 of 6
concurrent instances survived, and an atomic idempotent rule body still gave 0 of 6, because the
window is not the rule's to close.
What changed
The config records which whitelist the KB chose; the workflow module owns where it lands.
{onlist:<alias>}now resolves to the registry name, not a path. The compile-time refusal isuntouched —
registry.packed(name)still proves the list exists before anything is emitted.rule onlistinmap/starsoloandmap/chromapoutputstemp("onlists/{sample}/{name}.txt"),and each module's whitelist helper is a wildcard-taking input function.
RULES_THAT_MAY_OWN_A_SHARED_PATHis down toload_genomealone — the one exemption that has ameasurement behind it (6 of 6 across five trials).
That split is the point: with no location in the config, there is no second spelling free to drift
from the one the rule declares.
Why a per-sample copy, which the issue rejected
The issue killed this as "111 MB times the sample count". That arithmetic holds only if
temp()stops applying, and it does not — each copy is reclaimed after the job that read it, so the peak is
one copy per aligner job in flight, not one per sample. Measured: 6 794 880 barcodes, 115.5 MB,
6.5 s to expand, against a STAR job of tens of minutes.
It also does not reopen ADR-0015. That record rejects permanent duplication across compiles — a
third of a gigabyte sitting in run directories forever — and says nothing against per-job scratch a
rule reclaims.
What was deliberately not done
process exists, so no in-process guard is holding anything when it matters;
write_onlist_textalready writes a sibling partial and renames. The documented multi-job procedure also has the
operator pass
--nolock.actually does — declare no onlist rule, so the exposure was real by construction but unreached.
This change removes an exemption from an invariant that already holds, which needs no
demonstration.
load_genomeand its marker flag, the packed shipped form and the onlist registry are untouched.Tests
Two source-text assertions are gone; both claims moved onto the rendered plan, where a rename
cannot break them falsely and an indirection cannot pass them falsely — the producing verb, the
sample-scoped output path, and the reclamation line are all read off the same dry run that test
already spawned. The bd-rhapsody-enhanced case gains the ordering half of its three-whitelist claim:
a module free to build the paths is free to sort them, and a mis-paired round produces a plausible
matrix at exit 0.
WORKFLOW_VERSION→ 2026.8.25.Found on the way, not fixed here
e2e.pywrites"\n".join(onlist_files.values())as a whitelist file, but those values have beenregistry names since b1e9baf switched them from
_barcodes(packed)to[name]without updatingthe reader. So
kb e2e-costprices STARsolo runs whose--soloCBwhitelistis a one-line fileholding the string
3M-february-2018. Filed separately — out of scope here, and untouched by thischange.
🤖 Generated with Claude Code