Skip to content

bismark/align: intra-instance -p threading (replace --multicore fork model for Bowtie 2 + HISAT2)#12385

Open
FelixKrueger wants to merge 9 commits into
nf-core:masterfrom
FelixKrueger:bismark-align-p-threading
Open

bismark/align: intra-instance -p threading (replace --multicore fork model for Bowtie 2 + HISAT2)#12385
FelixKrueger wants to merge 9 commits into
nf-core:masterfrom
FelixKrueger:bismark-align-p-threading

Conversation

@FelixKrueger

@FelixKrueger FelixKrueger commented Jul 19, 2026

Copy link
Copy Markdown
Contributor

What & why

bismark/align auto-sets --multicore N. For Bowtie 2 that forks N workers, each loading the 2 (directional/pbat) or 4 (non-directional) converted-strand indexes — so peak RAM scales with N, and the cpus/3,cpus/5 divisor leaves cores idle.

This replaces the fork model with intra-instance threading: -p = task.cpus / n_instances (n = 2 directional/pbat, 4 non-directional, 1 for --combined_index). One index load per strand, all cpus used. Applies to Bowtie 2 and HISAT2; minimap2/rammap keep the legacy --multicore path (bismark can't route -p to them yet). An explicit --multicore/-p/--parallel in ext.args still short-circuits the auto-compute.

Bowtie 2 output stays byte-identical-p N --reorder is thread-invariant (verified 3-way against a single-core oracle). Benchmark (GRCh38, 2 M PE): wall −38 to −46 %, peak RAM −44 to −82 %.

Testing

nf-test green. The 5 existing tests are untouched — at the default cpus=2 no -p is emitted, so no snapshots regenerate; 4 new tests assert the emitted -p (from the report's options echo) at the 4-cpu CI ceiling.

Checklist

  • Description of changes with reason (above)
  • Tests added; nf-test passes in CI
  • No version bump / modules.json change

Replace the --multicore fork resource model with an intra-instance -p
divisor (-p = task.cpus / n_instances; n = 2 directional/pbat, 4
non-directional, 1 combined-index, 2 combined_index_parallel+non_dir)
for Bowtie 2 and HISAT2. minimap2/rammap keep the legacy --multicore
auto-compute unchanged.

Lower peak RAM (one index load per strand instance instead of M forks
each re-loading the index), fuller CPU use, and byte-identical Bowtie 2
output (-p N --reorder is thread-invariant). Emits -p only when
task.cpus/n >= 2, else omits it (faithful single-thread-per-instance).

Adds 4 nf-tests asserting -p is emitted (bowtie2 dir/non-dir, hisat2,
combined-index) via a dedicated cpus=8 config; existing tests unchanged
(cpus=2 emits no -p -> byte-identical, snapshots unchanged).
The -p-threading tests requested cpus=8 to force the divisor to emit -p,
but nf-core CI runners have 4 cpus and nextflow.enable.strict=true rejects
a request exceeding available cores (req 8 > avail 4) before the process
runs. Drop to cpus=4 (the runner ceiling): directional -> -p 2, HISAT2
directional -> -p 2, combined (n=1) -> -p 4. The non-directional case at
4/4=1 now asserts -p is OMITTED, exercising the omit-below-2 boundary.
Verified 9/9 nf-test pass on native amd64 with the 3.1.0 container.
@FelixKrueger
FelixKrueger requested a review from ewels July 20, 2026 14:53
Comment thread modules/nf-core/bismark/align/tests/main.nf.test
Comment thread modules/nf-core/bismark/align/main.nf
FelixKrueger added a commit to FelixKrueger/Bismark that referenced this pull request Jul 20, 2026
nf-core/modules#12385 review (mashehu) asks for the long-name flag. bismark's
-p was short-only (unlike bowtie2, which accepts -p/--threads); add the
--threads alias so downstream (nf-core module) can use the long name once a
bismark release with it ships. Behaviour-neutral: -p stays valid, no output
change (existing tests + perl-oracle unaffected).
@FelixKrueger

Copy link
Copy Markdown
Contributor Author

Thanks @mashehu! Two notes on these:

  1. Long name — good call. The catch is that the module invokes bismark (not bowtie2 directly), and bismark's CLI currently defines -p as short-only — no --threads alias, unlike bowtie2 itself — so bismark --threads N would fail to parse against the shipped 3.1.0 container. I've added a --threads alias to bismark (aligner: route -p to minimap2/rammap -t (thread-invariant, default -t 2) FelixKrueger/Bismark#1075); once a bismark release carrying it lands and this module bumps to that container, we can switch the flag here. Keeping -p for now so the module works against the current bismark:3.1.0.

  2. Test assertion — I've kept -p 2 --reorder rather than --threads 2: the assertion checks the alignment report, which echoes the underlying bowtie2 command. bismark always constructs that with -p internally regardless of how threads are passed to bismark, so --threads never appears in the report — the assertion has to stay -p.

FelixKrueger added a commit to FelixKrueger/Bismark that referenced this pull request Jul 20, 2026
… 2) (#1075)

* aligner: route -p to minimap2/rammap -t; kill 5-Base -t mangling (#1074)

minimap2_options renders -t from cli.bowtie_threads (Bismark -p), defaulting to
the faithful -t 2 when -p is absent — spike-confirmed minimap2 -t is thread-
invariant (byte-identical + order-preserving), so a bare --minimap2 stays
byte-identical to Perl. rammap-subprocess shares this clean-slate path; the
rammap in-process pool (inprocess_rammap_threads) now honors -p too
(precedence: -p > --multicore > capped available_parallelism).

The 5-Base path's fragile `.replace("-t 2", …)` — a SUBSTRING replace that
mangled `-t 20`→`-t 200` once the base carried a resolved `-t {p}` — is replaced
by a token-safe rewrite of the value after `-t`.

Closes #1074. Tests: 1427 lib tests pass incl. new -p-lift + C2 token-safe guard
+ rammap-pool -p cases; cargo fmt + clippy clean.

* aligner: add --threads long alias for -p (nf-core long-name convention)

nf-core/modules#12385 review (mashehu) asks for the long-name flag. bismark's
-p was short-only (unlike bowtie2, which accepts -p/--threads); add the
--threads alias so downstream (nf-core module) can use the long name once a
bismark release with it ships. Behaviour-neutral: -p stays valid, no output
change (existing tests + perl-oracle unaffected).
Comment thread modules/nf-core/bismark/align/tests/nextflow_pthreads.config
Presentation-only pass on the intra-instance -p threading change:
condense the aligner thread-model comment block and drop internal
plan jargon in the test config/assertions. No logic, test, or
snapshot changes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants