nf-core modules update#448
Draft
NicoDeVeaux wants to merge 7 commits into
Draft
Conversation
NicoDeVeaux
marked this pull request as draft
July 22, 2026 02:07
NicoDeVeaux
added a commit
to NicoDeVeaux/atacseq-nfcore
that referenced
this pull request
Jul 22, 2026
Use the release metadata fallback added in v3.0.1 so CI can install Nextflow while the nf-core metadata endpoint is empty. Update the changelog reference to PR nf-core#448 and remove the branch-local MultiQC deadlock entry.
|
Author
|
I'm going to keep this PR in draft mode while I work on getting nf-test to pass |
Bump every nf-core module and subworkflow via `nf-core modules update --all`
/ `nf-core subworkflows update --all` and reconcile the resulting
call-signature changes in first-party code. No intended behaviour change.
- Reconcile call sites in workflows/atacseq.nf and the affected
subworkflows/{local,nf-core}/*.
- Fix the ch_fasta_fai reference channel: add .first() so the genome
reference broadcasts to every sample instead of being consumed once.
- Consolidate singleton reference-channel plumbing.
- Remove the orphaned custom/getchromsizes module, replaced by
SAMTOOLS_FAIDX(..., get_sizes=true).
- Fix a terminal MultiQC hang: MultiQC must not publish to the `versions`
topic because its own input depends on that topic being resolved, which
creates a cycle that leaves the run hanging forever. Emit the version as
a named output instead, matching the nf-core/multiqc convention.
- Replace fragile '2>&1 | tr | sed;sed' version extraction in local modules
with concise single-command forms (still emitting topic: versions).
- Ignore a merge_markers false positive in the bowtie2 test snapshot: SAM
base-quality strings contain runs of '<' that trip the check.
Chromap BAMs carried no @rg header, so Picard 3.4.0 MarkDuplicates aborted with a NullPointerException (SAMRecord.getReadGroup() is null) once the module update bumped Picard from 3.1.1. Update the fastq_align_chromap subworkflow to d92e0f8, which adds PICARD_ADDORREPLACEREADGROUPS and an update_readgroups input, install the picard/addorreplacereadgroups module, enable read-group updating at the call site in workflows/atacseq.nf, and configure the process in conf/modules.config to mirror the existing BWA/Bowtie2 read-group mapping (RGID=meta.id technical id, RGSM=biological sample, RGPL=ILLUMINA, RGLB=meta.id, RGPU=1, RGCN=params.seq_center). The linked bam_sort_stats_samtools dependency tree also advanced samtools 1.23.1 -> 1.24 as part of the subworkflow update.
The module update bumped subread to 2.1.1, which strictly validates library pairing per BAM: it rejects single-end reads when run in paired mode (-p) and rejects paired-end reads when run in single-end mode. Consensus-peak quantification passes a mixed single-end/paired-end cohort to one SUBREAD_FEATURECOUNTS invocation, so 2.1.1 aborts with "No paired-end reads were detected in paired-end read library". Pin subread back to 2.0.1, which auto-detects pairing per BAM in a single invocation and reproduces the read/fragment counts the existing snapshots encode. This keeps the fix free of custom SE/PE-splitting and matrix-merging code. Remove the pin once mixed-library counting is handled explicitly (follow-up). The pin makes the local module diverge from its registry git_sha, so it is recorded as a tracked nf-core module patch (container + environment.yml) for `nf-core modules lint` check_local_copy.
… order The consensus BAM list is assembled from an unordered channel collect, so the featureCounts invocation received its BAMs (and emitted its count columns) in a run- and host-dependent order. The counts were identical, but the output file's byte content - and therefore its snapshot md5 - varied between runs, making the consensus featureCounts snapshot non-reproducible. Sort the BAM list by filename before featureCounts so the column order (and the output file) is deterministic and identical on any x86_64 host, matching CI.
…ates metrics Regenerate all 7 pipeline nf-test snapshots against the updated modules (subread 2.0.1 pin, samtools bumps, chromap read-group path, deterministic consensus featureCounts column order). The previous snapshots were ~2 months stale, from before the module update. Generated on GitHub-hosted ubuntu-latest x86_64 runners via the update-snapshots workflow, so container digests and architecture match CI. Also extend .nftignore: Picard MarkDuplicates metrics headers carry a per-run "Started on: <timestamp>" line, so their md5 is never reproducible. The existing pattern only covered the default filename (*.MarkDuplicates.metrics.txt), but this pipeline renames them via ext.prefix to *.mkD.sorted.metrics.txt (library) and *.mRp.clN.sorted.metrics.txt (replicate). These were the only non-deterministic files left once the consensus featureCounts inputs were sorted.
Merge the nf-core/tools 4.0.3 template output (the same content as nf-core#449) so this branch lints clean against the current template rather than carrying files_unchanged exemptions for the pre-existing drift in .github/PULL_REQUEST_TEMPLATE.md and the two logo PNGs. Keeps the branch-local .nf-core.yml entries (bowtie2 snapshot merge_markers false positive, bamtools filter config defaults) and regenerates ro-crate-metadata.json. Drop this commit if nf-core#449 lands on dev first.
NicoDeVeaux
force-pushed
the
nf-core-modules-update
branch
from
July 26, 2026 00:51
1513428 to
e1dc032
Compare
|
Add entries for the Chromap read-group fix, the subread 2.0.1 pin and the deterministic consensus featureCounts column order. The template 4.0.3 bump is left to nf-core#449.
Author
4 tasks
NicoDeVeaux
added a commit
to NicoDeVeaux/atacseq-nfcore
that referenced
this pull request
Jul 26, 2026
Restore subread to the registry version and drop the nf-core patch record. The pin was the stopgap for mixed single-end/paired-end consensus counting; the following commits handle mixed cohorts explicitly, so it is no longer needed. Reverts the pin introduced in nf-core#448.
NicoDeVeaux
added a commit
to NicoDeVeaux/atacseq-nfcore
that referenced
this pull request
Jul 26, 2026
The SE/PE split replaced the single sorted BAM list from nf-core#448 with two per-endedness batches plus a merge step, each assembled from an unordered channel collect. Sort all three by filename so the featureCounts column order, the merged count matrix and its snapshot md5 stay reproducible across runs and hosts, preserving the guarantee nf-core#448 established.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Updates all nf-core modules and subworkflows to their latest upstream versions (except
subread, see details below)and reconciles the resulting call-signature changes across the pipeline. No intended change in pipeline behavior. This implements Issue 447
What changed and why
Updated all nf-core modules and subworkflows to their latest available revisions using:
The resulting main.nf, meta.yml, and nf-test snapshot changes were retained, and the required ATAC-seq call sites were reconciled with the updated input and output signatures.
Reconciled call sites in
workflows/atacseq.nfand the affectedsubworkflows/local/*andsubworkflows/nf-core/*to the updated signatures.Updated versions topic, but kept MultiQC version reporting as a named output rather than publishing it
to the versions topic, avoiding a self-dependency because MultiQC consumes
the collated version information as an input.
Add read groups to Chromap alignments, which Picard 3.4.0
MarkDuplicatesnow requires.Pin
subreadto 2.0.1 so consensus peak counting keeps supporting mixed single-end/paired-end cohorts in a singlefeatureCountsinvocation.Sort the consensus
featureCountsBAM inputs so the count matrix column order is deterministic.Verification:
Real-data A/B testing against dev was completed on AWS Batch using two
four-sample datasets.
72-process DAG and executed 185 tasks. MULTIQC completed as the terminal
node in both runs, confirming resolution of the previous topic-channel
deadlock on a second dataset. Scientific outputs are near-identical. The
only differences are ±2 peaks in the two lowest-depth libraries, which
have only single- or double-digit peak counts. Higher-signal samples are
bit-identical.
every per-library and per-replicate peak count and FRiP score matches to
full precision. Per-library peak counts are 30,438 / 35,854 / 36,551 /
51,633, with FRiP scores of 0.216448 / 0.230010 / 0.198316 / 0.250963.
The merged-library and merged-replicate consensus peak sets are also
identical at 74,895 and 115,880 peaks, respectively.
PR checklist
nf-core lint).nextflow run . -profile test,docker --outdir <OUTDIR>).docs/usage.mdis updated.docs/output.mdis updated.CHANGELOG.mdis updated.README.mdis updated (including new tool citations and authors/contributors).