Google Cloud Batch

The google-batch executor runs pipeline tasks using Google Cloud Batch, a managed compute service that runs containerized workloads in Google Cloud. It submits each task as a Cloud Batch job.

To enable this executor, set process.executor = 'google-batch' in the nextflow.config file. You can launch the pipeline from a local machine or a Google Compute Engine VM. Every pipeline process must specify a Docker image using the container directive. The pipeline work directory must be located in a Google Cloud Storage bucket.

Use the following process directives to control resource requests and other job characteristics:

• accelerator
• container
• containerOptions
• cpus
• disk
• machineType
• memory
• resourcelabels
• time

The following hints are supported:

• scheduling.spotAttempts: Run a process on Spot for its first N attempts, then automatically fall back to on-demand (STANDARD) for later attempts. The value is a positive integer N: attempts 1 to N request a Spot VM, attempts after N request on-demand. The attempt count is the greater of the task's execution attempt and its submit attempt, so the fallback is triggered both by a mid-run Spot reclaim and by a failure to obtain a Spot VM (the latter in combination with the maxSubmitAwait directive). Requires errorStrategy 'retry' with maxRetries set high enough to reach the on-demand attempts. For example:

  process EXAMPLE {
      errorStrategy 'retry'
      maxRetries 3
      hints 'scheduling.spotAttempts': '2'   // Spot for attempts 1 and 2, on-demand from attempt 3
  }

  This key may be used as-is or with the google-batch/ prefix to restrict it to this executor.

  note

  scheduling.spotAttempts is distinct from the google.batch.maxSpotAttempts config option, despite the similar name. google.batch.maxSpotAttempts sets the number of times Cloud Batch retries a job internally after a Spot reclaim, and each retry requests a Spot VM again; it applies to every process in the run. scheduling.spotAttempts instead changes the provisioning model across Nextflow-level attempts, and is set per process. The two can be combined.

See Cloud Batch for further configuration details.