Visium HD integration

.github/workflows/build_container.yml

@@ -1,13 +1,16 @@
-name: Build R container
+name: Build R containers
 
-# The image tag is read from conf/containers.config, so that single line is both what the
-# pipeline pulls and what this workflow publishes -- the two cannot drift.
-# Runs on pushes to a feature branch, whenever the container changes.
+# The image tag of the container is read from conf/containers.config, so that the tag is
+# what the pipeline pulls and what this workflow publishes. Each container is built independently,
+# so updating one tag in conf/containers.config rebuilds only that image:
+#   feature branch - rebuilds its tag on every push, unless main or devel is pinned to that tag
+#   main or devel  - builds only a tag that has never been published, so most merges are a no-op
+#   manual run     - 'force' rebuilds and overwrites the tag on any branch
 
 on:
   push:
     paths:
-      - 'containers/r/**'
+      - 'containers/**'
       - 'conf/containers.config'
       - '.github/workflows/build_container.yml'
   workflow_dispatch:
@@ -22,81 +25,130 @@ concurrency:
   cancel-in-progress: true
 
 jobs:
-  build:
-    name: Build and publish
+  discover:
+    name: Select containers to build
     runs-on: ubuntu-latest
     permissions:
       contents: read
-      packages: write
+      packages: read
+    outputs:
+      containers: ${{ steps.select.outputs.containers }}
 
     steps:
       - uses: actions/checkout@v4
 
-      - name: Resolve image from conf/containers.config
-        id: image
+      - uses: docker/login-action@v3
+        with:
+          registry: ghcr.io
+          username: ${{ github.actor }}
+          password: ${{ secrets.GITHUB_TOKEN }}
+
+      - name: Select containers to build
+        id: select
         run: |
-          IMAGE=$(sed -n "s|.*withLabel: *'r'.*container *= *\"\([^\"]*\)\".*|\1|p" conf/containers.config)
-          if [ -z "$IMAGE" ]; then
-            echo "::error::could not parse the 'r' container from conf/containers.config"
-            exit 1
+          RELEASED=false
+          case "${{ github.ref_name }}" in main|devel) RELEASED=true ;; esac
+
+          # Each base branch is fetched into its own ref so both stay readable in the loop
+          BASES=""
+          for BASE in devel main; do
+            if git fetch --no-tags --depth=1 origin "$BASE:refs/base/$BASE" 2>/dev/null; then
+              BASES="$BASES refs/base/$BASE"
+            fi
+          done
+
+          # devel is at or ahead of main, so it is the baseline for "did this branch edit a container"
+          DEVEL=refs/base/devel
+
+          # a container is only rebuilt by a push that touches it, so the previous push is the
+          # baseline for that. devel is the fallback when that commit cannot be fetched, which is
+          # the case on the first push of a branch, on a force push and on a manual run
+          PUSH_BASE=$DEVEL
+          BEFORE="${{ github.event.before }}"
+          if [ -n "$BEFORE" ] && git fetch --no-tags --depth=1 origin "$BEFORE" 2>/dev/null; then
+            PUSH_BASE=$BEFORE
           fi
-          case "$IMAGE" in
-            ghcr.io/goekelab/*) ;;
-            *) echo "::error::refusing to push outside ghcr.io/goekelab: $IMAGE"; exit 1 ;;
-          esac
-          echo "image=$IMAGE" >> "$GITHUB_OUTPUT"
-          echo "Resolved container: $IMAGE"
-
-      - name: Refuse to overwrite a tag main or devel is pinned to
-        if: github.ref_name != 'main' && github.ref_name != 'devel'
-        run: |
-          IMAGE="${{ steps.image.outputs.image }}"
-          for BASE in main devel; do
-            git fetch --no-tags --depth=1 origin "$BASE" 2>/dev/null || continue
-            PINNED=$(git show "FETCH_HEAD:conf/containers.config" 2>/dev/null \
-              | sed -n "s|.*withLabel: *'r'.*container *= *\"\([^\"]*\)\".*|\1|p")
-            if [ "$IMAGE" = "$PINNED" ]; then
-              echo "::error::$BASE is pinned to $IMAGE -- bump the tag in conf/containers.config before rebuilding it"
+
+          # Every containers/<label>/Dockerfile is an image whose <label> is its process label
+          : > selected.txt
+          for DOCKERFILE in containers/*/Dockerfile; do
+            LABEL=$(basename "$(dirname "$DOCKERFILE")")
+            IMAGE=$(sed -n "s|.*withLabel: *'$LABEL'.*container *= *\"\([^\"]*\)\".*|\1|p" conf/containers.config)
+
+            if [ -z "$IMAGE" ]; then
+              echo "::error::could not parse the '$LABEL' container from conf/containers.config"
               exit 1
             fi
+            case "$IMAGE" in
+              ghcr.io/goekelab/*) ;;
+              *) echo "::error::refusing to push outside ghcr.io/goekelab: $IMAGE"; exit 1 ;;
+            esac
+
+            EDITED_ON_BRANCH=false
+            if ! git diff --quiet "$DEVEL" HEAD -- "containers/$LABEL"; then
+              EDITED_ON_BRANCH=true
+            fi
+
+            EDITED_BY_PUSH=false
+            if ! git diff --quiet "$PUSH_BASE" HEAD -- "containers/$LABEL"; then
+              EDITED_BY_PUSH=true
+            fi
+
+            # A tag main or devel is pinned to is immutable, so an edit needs a tag bump first
+            if [ "$RELEASED" = false ] && [ "$EDITED_ON_BRANCH" = true ]; then
+              for REF in $BASES; do
+                PINNED=$(git show "$REF:conf/containers.config" 2>/dev/null \
+                  | sed -n "s|.*withLabel: *'$LABEL'.*container *= *\"\([^\"]*\)\".*|\1|p")
+                if [ "$IMAGE" = "$PINNED" ]; then
+                  echo "::error::${REF#refs/base/} is pinned to $IMAGE -- bump the tag in conf/containers.config before rebuilding it"
+                  exit 1
+                fi
+              done
+            fi
+
+            # An unpublished tag is always built, since the pipeline cannot pull what is not there
+            if [ "${{ inputs.force }}" != "true" ] && docker manifest inspect "$IMAGE" >/dev/null 2>&1; then
+              if [ "$RELEASED" = true ] || [ "$EDITED_BY_PUSH" = false ]; then
+                echo "::notice::$IMAGE is already published and unchanged -- skipping $LABEL"
+                continue
+              fi
+            fi
+
+            echo "$LABEL $IMAGE" >> selected.txt
           done
 
+          CONTAINERS=$(jq -R -s -c 'split("\n") | map(select(length > 0) | split(" ")) | map({label: .[0], image: .[1]})' < selected.txt)
+          echo "containers=$CONTAINERS" >> "$GITHUB_OUTPUT"
+          echo "Building: $CONTAINERS"
+
+  build:
+    name: Build and publish (${{ matrix.container.label }})
+    needs: discover
+    if: needs.discover.outputs.containers != '[]'
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+      packages: write
+    strategy:
+      fail-fast: false
+      matrix:
+        container: ${{ fromJson(needs.discover.outputs.containers) }}
+
+    steps:
+      - uses: actions/checkout@v4
+
       - uses: docker/login-action@v3
         with:
           registry: ghcr.io
           username: ${{ github.actor }}
           password: ${{ secrets.GITHUB_TOKEN }}
 
-      - name: Decide whether to build
-        id: decide
-        run: |
-          IMAGE="${{ steps.image.outputs.image }}"
-          if docker manifest inspect "$IMAGE" >/dev/null 2>&1; then EXISTS=true; else EXISTS=false; fi
-
-          if [ "${{ inputs.force }}" = "true" ]; then
-            BUILD=true
-          elif [ "${{ github.ref_name }}" = "main" ] || [ "${{ github.ref_name }}" = "devel" ]; then
-            # Released tags are immutable: the merge that brought the bump in has already
-            # published it from the feature branch, so there is nothing left to do
-            [ "$EXISTS" = "false" ] && BUILD=true || BUILD=false
-          else
-            # Feature branch: the tag is unreleased (guarded above), so rebuild freely
-            BUILD=true
-          fi
-
-          echo "build=$BUILD" >> "$GITHUB_OUTPUT"
-          if [ "$BUILD" = "false" ]; then
-            echo "::notice::$IMAGE is already published -- skipping build"
-          fi
-
       - uses: docker/setup-buildx-action@v3
-        if: steps.decide.outputs.build == 'true'
 
       - uses: docker/build-push-action@v6
-        if: steps.decide.outputs.build == 'true'
         with:
-          context: containers/r
+          context: containers/${{ matrix.container.label }}
           push: true
-          tags: ${{ steps.image.outputs.image }}
-          cache-from: type=gha
-          cache-to: type=gha,mode=max
+          tags: ${{ matrix.container.image }}
+          cache-from: type=gha,scope=${{ matrix.container.label }}
+          cache-to: type=gha,mode=max,scope=${{ matrix.container.label }}

.github/workflows/smoke_test.yml

@@ -11,7 +11,7 @@ concurrency:
 
 jobs:
   smoke_test:
-    name: "${{ matrix.profile }} (NF ${{ matrix.nextflow_version }})${{ matrix.quantification_mode && format(' [{0}]', matrix.quantification_mode) || '' }}${{ matrix.bam_only && ' [bam_only]' || '' }}"
+    name: "${{ matrix.profile }} (NF ${{ matrix.nextflow_version }})${{ matrix.quantification_mode && format(' [{0}]', matrix.quantification_mode) || '' }}${{ matrix.bam_only && ' [bam_only]' || '' }}${{ matrix.extra_args && format(' [{0}]', matrix.extra_args) || '' }}"
     runs-on: ubuntu-latest
     strategy:
       matrix:
@@ -21,16 +21,24 @@ jobs:
           - {nextflow_version: "26.04.0", profile: test_sc_bam}
           - {nextflow_version: "26.04.0", profile: test_sc_multi}
           - {nextflow_version: "26.04.0", profile: test_visium}
+          - {nextflow_version: "26.04.0", profile: test_visium_hd}
           - {nextflow_version: "26.04.0", profile: test_custom}
-          # latest-stable - all profiles (default EM_clusters)
+          - {nextflow_version: "26.04.0", profile: test_sc_quant_data}
+          - {nextflow_version: "26.04.0", profile: test_visium_hd_quant_data}
+          # latest-stable - all profiles (default clusteredEM)
           - {nextflow_version: "latest-stable", profile: test_sc_fastq}
           - {nextflow_version: "latest-stable", profile: test_sc_bam}
           - {nextflow_version: "latest-stable", profile: test_sc_multi}
           - {nextflow_version: "latest-stable", profile: test_visium}
+          - {nextflow_version: "latest-stable", profile: test_visium_hd}
           - {nextflow_version: "latest-stable", profile: test_custom}
+          - {nextflow_version: "latest-stable", profile: test_sc_quant_data}
+          - {nextflow_version: "latest-stable", profile: test_visium_hd_quant_data}
           # latest-stable - extra parameter variants
           - {nextflow_version: "latest-stable", profile: test_sc_fastq, quantification_mode: EM}
           - {nextflow_version: "latest-stable", profile: test_sc_multi, quantification_mode: EM}
+          - {nextflow_version: "latest-stable", profile: test_visium_hd, quantification_mode: EM}
+          - {nextflow_version: "latest-stable", profile: test_visium_hd, extra_args: "--banksy false"}
           - {nextflow_version: "latest-stable", profile: test_sc_fastq, quantification_mode: no_quant}
           - {nextflow_version: "latest-stable", profile: test_sc_fastq, bam_only: true}
     steps:
@@ -42,4 +50,5 @@ jobs:
         run: |
           nextflow run . -profile test_base,${{ matrix.profile }},docker \
             ${{ matrix.quantification_mode && format('--quantification_mode {0}', matrix.quantification_mode) || '' }} \
-            ${{ matrix.bam_only && '--bam_only' || '' }}
+            ${{ matrix.bam_only && '--bam_only' || '' }} \
+            ${{ matrix.extra_args || '' }}

CHANGELOG.md

@@ -4,6 +4,32 @@ This file contains all notable changes to Bambu-Pipe.
 
 ---
 
+## [v0.10.0] - 2026-08-05
+
+### Added
+- Visium HD workflow (`--visium_hd`), run as a single sample from a Spaceranger-aligned, barcode-tagged BAM
+  - Transcript discovery and read-to-transcript assignment at the native 2 µm resolution, with counts aggregated to every bin listed in `--bins`
+  - `--barcode_mappings` for the Spaceranger `barcode_mappings.parquet`, used to assign 2 µm spots to bins
+  - Out-of-tissue reads filtered from the BAM using the 2 µm `tissue_positions.parquet`
+  - Spatially aware clustering with Banksy (`--banksy`, `--banksy_lambda`, `--banksy_k_geom`), or gene expression alone
+  - `--clustering_bin` to select the resolution to cluster at; cluster labels are expanded back to 2 µm spots for quantification
+  - Spot-level quantification at every resolution under `--quantification_mode EM`
+  - `test_visium_hd` smoke test profile with synthetic example data
+- `--manual_clustering` to restart the pipeline from cluster assignments generated outside the pipeline, for both standard and Visium HD runs
+- `test_sc_quant_data` and `test_visium_hd_quant_data` smoke test profiles covering the manual clustering restart
+- Self-hosted `bambu` and `seurat` container images published to `ghcr.io/goekelab`, built by the `build_container.yml` GitHub Actions workflow
+- Shared R helpers in `bin/` for transcript discovery, Seurat object creation, count saving, and cluster mapping
+
+### Changed
+- Renamed `--resolution` to `--seurat_resolution`
+- Renamed the `--quantification_mode` option `EM_clusters` to `clusteredEM`, matching the `bambu.singlecell` API
+- `quant_data.rds` and `extended_annotations.rds` are now always published to `intermediate_R/`, so a manual clustering run can restart from them
+- Seurat objects are built from the published count directories and Bambu's `colData` instead of the `SummarizedExperiment`
+- `clusters.rds` is now a named vector of `id -> cluster` label, replacing the per-sample list of `CompressedCharacterList`
+- Cluster-level quantification moved into a single module shared by the standard and Visium HD workflows
+- Restructured modules into `standard/`, `visium_hd/`, and `shared/` directories
+- Smoke tests now run on pull request and manual dispatch only, with in-progress runs cancelled on a new push
+
 ## [v0.9.1] - 2026-05-20
 
 ### Added