Skip to content

the whitelist lands under the sample that reads it, so no planned job owns a path a second instance deletes - #492

Merged
lhqing merged 2 commits into
mainfrom
fix/488-whitelist-owns-no-shared-path
Aug 22, 2026
Merged

lhqing merged 2 commits into
mainfrom
fix/488-whitelist-owns-no-shared-path

Conversation

@lhqing

@lhqing lhqing commented Aug 22, 2026

Copy link
Copy Markdown
Contributor

Closes #488.

The barcode whitelist was the last rule output a whole deposit shared. Snakemake removes an output
before running the job that produces it, and reclaims a temp() output when its last consumer in
that instance
finishes — so a second instance over one results directory raced the first's
removal, not its write. That is the hazard class #478 measured on the index rule: 0 of 6
concurrent instances survived, and an atomic idempotent rule body still gave 0 of 6, because the
window is not the rule's to close.

What changed

The config records which whitelist the KB chose; the workflow module owns where it lands.

  • {onlist:<alias>} now resolves to the registry name, not a path. The compile-time refusal is
    untouched — registry.packed(name) still proves the list exists before anything is emitted.
  • rule onlist in map/starsolo and map/chromap outputs temp("onlists/{sample}/{name}.txt"),
    and each module's whitelist helper is a wildcard-taking input function.
  • RULES_THAT_MAY_OWN_A_SHARED_PATH is down to load_genome alone — the one exemption that has a
    measurement behind it (6 of 6 across five trials).

That split is the point: with no location in the config, there is no second spelling free to drift
from the one the rule declares.

Why a per-sample copy, which the issue rejected

The issue killed this as "111 MB times the sample count". That arithmetic holds only if temp()
stops applying, and it does not — each copy is reclaimed after the job that read it, so the peak is
one copy per aligner job in flight, not one per sample. Measured: 6 794 880 barcodes, 115.5 MB,
6.5 s to expand, against a STAR job of tens of minutes.

It also does not reopen ADR-0015. That record rejects permanent duplication across compiles — a
third of a gigabyte sitting in run directories forever — and says nothing against per-job scratch a
rule reclaims.

What was deliberately not done

  • No lock, and no skip-if-present guard on the writer. The deletion happens before the rule's
    process exists, so no in-process guard is holding anything when it matters; write_onlist_text
    already writes a sibling partial and renames. The documented multi-job procedure also has the
    operator pass --nolock.
  • No six-instance reproduction. The two plate modules — where a sharded run is what anyone
    actually does — declare no onlist rule, so the exposure was real by construction but unreached.
    This change removes an exemption from an invariant that already holds, which needs no
    demonstration.
  • load_genome and its marker flag, the packed shipped form and the onlist registry are untouched.

Tests

Two source-text assertions are gone; both claims moved onto the rendered plan, where a rename
cannot break them falsely and an indirection cannot pass them falsely — the producing verb, the
sample-scoped output path, and the reclamation line are all read off the same dry run that test
already spawned. The bd-rhapsody-enhanced case gains the ordering half of its three-whitelist claim:
a module free to build the paths is free to sort them, and a mis-paired round produces a plausible
matrix at exit 0.

WORKFLOW_VERSION → 2026.8.25.

Found on the way, not fixed here

e2e.py writes "\n".join(onlist_files.values()) as a whitelist file, but those values have been
registry names since b1e9baf switched them from _barcodes(packed) to [name] without updating
the reader. So kb e2e-cost prices STARsolo runs whose --soloCBwhitelist is a one-line file
holding the string 3M-february-2018. Filed separately — out of scope here, and untouched by this
change.

🤖 Generated with Claude Code

lhqing and others added 2 commits August 22, 2026 14:59
… owns a path a second instance deletes

The config records the registry name the KB chose and the consuming module builds
the path, which is what lets that path be one no two jobs share: one owner for the
location, so config and rule cannot disagree. `rule onlist` keeps temp(), keeps its
verb, keeps no container, and gains a sample scope in both modules.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…s where the copy lands and when it goes

RULES_THAT_MAY_OWN_A_SHARED_PATH is down to the load flag, the one path measured
safe to race. The two claims a module's source text used to carry -- temp() and the
producing verb -- move onto the rendered plan, where a rename cannot break them
falsely and an indirection cannot pass them falsely, and the split-pool dry run
gains the three paths in CB-position order as one argv token.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Copilot AI lite review requested due to automatic review settings August 22, 2026 19:02

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@lhqing
lhqing merged commit f26259a into main Aug 22, 2026
6 checks passed
@lhqing
lhqing deleted the fix/488-whitelist-owns-no-shared-path branch August 22, 2026 19:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The barcode whitelist is a shared deletable path, so two instances over one directory race it

2 participants