bismark/align: intra-instance -p threading (replace --multicore fork model for Bowtie 2 + HISAT2)#12385
bismark/align: intra-instance -p threading (replace --multicore fork model for Bowtie 2 + HISAT2)#12385FelixKrueger wants to merge 9 commits into
Conversation
Replace the --multicore fork resource model with an intra-instance -p divisor (-p = task.cpus / n_instances; n = 2 directional/pbat, 4 non-directional, 1 combined-index, 2 combined_index_parallel+non_dir) for Bowtie 2 and HISAT2. minimap2/rammap keep the legacy --multicore auto-compute unchanged. Lower peak RAM (one index load per strand instance instead of M forks each re-loading the index), fuller CPU use, and byte-identical Bowtie 2 output (-p N --reorder is thread-invariant). Emits -p only when task.cpus/n >= 2, else omits it (faithful single-thread-per-instance). Adds 4 nf-tests asserting -p is emitted (bowtie2 dir/non-dir, hisat2, combined-index) via a dedicated cpus=8 config; existing tests unchanged (cpus=2 emits no -p -> byte-identical, snapshots unchanged).
The -p-threading tests requested cpus=8 to force the divisor to emit -p, but nf-core CI runners have 4 cpus and nextflow.enable.strict=true rejects a request exceeding available cores (req 8 > avail 4) before the process runs. Drop to cpus=4 (the runner ceiling): directional -> -p 2, HISAT2 directional -> -p 2, combined (n=1) -> -p 4. The non-directional case at 4/4=1 now asserts -p is OMITTED, exercising the omit-below-2 boundary. Verified 9/9 nf-test pass on native amd64 with the 3.1.0 container.
nf-core/modules#12385 review (mashehu) asks for the long-name flag. bismark's -p was short-only (unlike bowtie2, which accepts -p/--threads); add the --threads alias so downstream (nf-core module) can use the long name once a bismark release with it ships. Behaviour-neutral: -p stays valid, no output change (existing tests + perl-oracle unaffected).
|
Thanks @mashehu! Two notes on these:
|
… 2) (#1075) * aligner: route -p to minimap2/rammap -t; kill 5-Base -t mangling (#1074) minimap2_options renders -t from cli.bowtie_threads (Bismark -p), defaulting to the faithful -t 2 when -p is absent — spike-confirmed minimap2 -t is thread- invariant (byte-identical + order-preserving), so a bare --minimap2 stays byte-identical to Perl. rammap-subprocess shares this clean-slate path; the rammap in-process pool (inprocess_rammap_threads) now honors -p too (precedence: -p > --multicore > capped available_parallelism). The 5-Base path's fragile `.replace("-t 2", …)` — a SUBSTRING replace that mangled `-t 20`→`-t 200` once the base carried a resolved `-t {p}` — is replaced by a token-safe rewrite of the value after `-t`. Closes #1074. Tests: 1427 lib tests pass incl. new -p-lift + C2 token-safe guard + rammap-pool -p cases; cargo fmt + clippy clean. * aligner: add --threads long alias for -p (nf-core long-name convention) nf-core/modules#12385 review (mashehu) asks for the long-name flag. bismark's -p was short-only (unlike bowtie2, which accepts -p/--threads); add the --threads alias so downstream (nf-core module) can use the long name once a bismark release with it ships. Behaviour-neutral: -p stays valid, no output change (existing tests + perl-oracle unaffected).
Presentation-only pass on the intra-instance -p threading change: condense the aligner thread-model comment block and drop internal plan jargon in the test config/assertions. No logic, test, or snapshot changes.
What & why
bismark/alignauto-sets--multicore N. For Bowtie 2 that forks N workers, each loading the 2 (directional/pbat) or 4 (non-directional) converted-strand indexes — so peak RAM scales with N, and thecpus/3,cpus/5divisor leaves cores idle.This replaces the fork model with intra-instance threading:
-p = task.cpus / n_instances(n = 2 directional/pbat, 4 non-directional, 1 for--combined_index). One index load per strand, all cpus used. Applies to Bowtie 2 and HISAT2; minimap2/rammap keep the legacy--multicorepath (bismark can't route-pto them yet). An explicit--multicore/-p/--parallelinext.argsstill short-circuits the auto-compute.Bowtie 2 output stays byte-identical —
-p N --reorderis thread-invariant (verified 3-way against a single-core oracle). Benchmark (GRCh38, 2 M PE): wall −38 to −46 %, peak RAM −44 to −82 %.Testing
nf-testgreen. The 5 existing tests are untouched — at the defaultcpus=2no-pis emitted, so no snapshots regenerate; 4 new tests assert the emitted-p(from the report's options echo) at the 4-cpu CI ceiling.Checklist
nf-testpasses in CImodules.jsonchange