Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
65 changes: 65 additions & 0 deletions hcls/nextflow-batch/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
# Nextflow on Google Batch

This guide provides instructions for running the Nextflow [`rnaseq-nf`](https://github.com/nextflow-io/rnaseq-nf) pipeline sample on Google Cloud Batch.

## References

* [Nextflow Installation](https://docs.seqera.io/nextflow/install)
* [Nextflow on Google Cloud Batch](https://docs.cloud.google.com/batch/docs/nextflow)

## Reference Architecture

<img src="img/nextflow-w-batch-reference.png" alt="Nextflow on Google Cloud Batch" width="800">

## Prerequisites

1. **Install Nextflow:**
Follow the instructions from the [Nextflow installation guide](https://docs.seqera.io/nextflow/install). User can operate Nextflow at "anywhere" including Cloud Shell, Cloudtop or local machine.

2. **Enable Google Cloud APIs:**
Ensure you have enabled the necessary APIs in your GCP project:
* Batch API (`batch.googleapis.com`)
* Compute Engine API (`compute.googleapis.com`)
* Cloud Storage API (`storage.googleapis.com`)

3. **Google Cloud Storage (GCS) Bucket:**
Create a GCS bucket to store the pipeline's work directory and output files.

4. **Authentication:**
Authenticate your environment with Google Cloud:
```bash
gcloud auth login
gcloud auth application-default login
```

## Configuration

User can leverage the `nextflow.config` file in this repo to configure Nextflow to use Google Batch as the executor

Please make sure updating the following parameters according to your GCP setup.

```bash
params.bucketname = '<your bucket name>'
params.gcp_project = '<your gcp project>'
params.network_name = 'default'
params.subnet_name = 'default'
params.location = 'us-central1'
```

## Clone the sample rnaseq-nf git

git clone https://github.com/nextflow-io/rnaseq-nf.git

## Running the sample rnaseq-nf Pipeline

Once you have your `nextflow.config` file set up, you can run the `rnaseq-nf` sample pipeline with the following command:

```bash
./nextflow run rnaseq-nf -profile google-batch
```

Nextflow will automatically parse the config in the current directory, provision the necessary resources on Google Cloud Batch, and execute the pipeline steps.

## Monitoring

You can monitor the status and execution of your pipeline jobs directly in the Google Cloud Batch Jobs Console.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
52 changes: 52 additions & 0 deletions hcls/nextflow-batch/nextflow.config
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
manifest {
description = 'Proof of concept of a RNA-seq pipeline implemented with Nextflow'
author = 'Paolo Di Tommaso'
nextflowVersion = '>=25.10.0'
}

/*
* publish settings
*/
workflow.output.mode = 'copy'

/*
* execution profiles for different environments
*/
process {
// Default resources for all processes
memory = '16 GB'
machineType = 'n2-standard-8' // VM type for cloud executors like Google Batch

// Specific configurations for the INDEX process
withName: 'INDEX' {
memory = '16 GB'
machineType = 'n2-standard-8'
cpus = 4
}
}

profiles {
'google-batch' {
params.transcriptome = 'gs://rnaseq-nf/data/ggal/transcript.fa'
params.reads = 'gs://rnaseq-nf/data/ggal/gut_{1,2}.fq'
params.multiqc = 'gs://rnaseq-nf/multiqc'
params.bucketname = '<your bucket name>'
params.gcp_project = '<your gcp project>'
params.network_name = 'default'
params.subnet_name = 'default'
params.location = 'us-central1'

process.executor = 'google-batch'
process.container = 'docker.io/nextflow/rnaseq-nf:v1.3.1'

workDir = "gs://${params.bucketname}/scratch"
google.location = 'us-central1'
google.project = params.gcp_project
google.batch.allowedLocations = ['regions/us-central1']
/*
* adding network information if not using default network
*/
google.batch.network = "global/networks/${params.network_name}"
google.batch.subnetwork = "regions/${params.location}/subnetworks/${params.subnet_name}"
}
}