Nextflow

Key takeaways

Nextflow belongs to computer science; place it in that map before memorising details.
Learn the definition first, then one example that makes the definition concrete.
Connect Nextflow to a quantity you can measure, compute or draw — that is where exam questions come from.
Reproduce the core statement of Nextflow from memory before moving on to harder problems.

Reference excerpt

Nextflow is a scientific workflow system predominantly used for bioinformatic data analysis. It establishes standards for programmatically creating a series of dependent computational steps and facilitates their execution on various local and cloud resources.

Purpose Many scientific data analyses require a significant amount of sequential processing steps. Custom scripts may suffice when developing new methods or infrequently running particular analyses, but scale poorly to complex task successions or many samples. Scientific workflow systems like Nextflow allow formalizing an analysis as a data analysis pipeline. Pipelines, also known as workflows, specify the order and conditions of computing steps. They are accomplished by special purpose programs, so-called workflow executors, which ensure predictable and reproducible behavior in various computing environments. Workflow systems also provide built-in solutions to common challenges of workflow development, such as the application to multiple samples, the validation of input and intermediate results, conditional execution of steps, error handling, and report generation. Advanced features of workflow systems may also include scheduling capabilities, graphical user interfaces for monitoring workflow executions, and the management of dependencies by containerizing the whole workflow or its components. Typically, scientific workflow systems initially present a steep learning challenge as all their features and complexities are built on in addition to the actual analysis. However, the standards and abstraction imposed by workflow systems ultimately improve the traceability of analysis steps, which is particularly relevant when collaborating on pipeline development, as is customary in scientific settings.

Characteristics

Specification of workflows In Nextflow, pipelines are constructed from individual processes that work in parallel to perform computational tasks. Each process is defined with input requirements and output declarations. Instead of running in a fixed sequence, a process starts executing when all its input requirements are fulfilled. By specifying the output of one process as the input of another, a logical and sequential connection between processes is established. This reactive implementation is a key design pattern of Nextflow and is also known as the functional dataflow model. Processes and entire workflows are programmed in a domain-specific language (DSL) which is provided by Nextflow which is based on Apache Groovy. While Nextflow's DSL is used to declare the workflow logic, developers can use their scripting language of choice within a process and mix multiple languages in a workflow. It is also possible to port existing scripts and workflows to Nextflow. Supported scripting languages include bash, csh, ksh, Python, Ruby, and R. Any scripting language that uses the standard Unix shebang declaration (#!/bin/bash) is compatible with Nextflow. Below is an example of a workflow consisting of only one process:

To enable easy collaboration on workflows, Nextflow natively support for source-code management systems and DevOps platforms including GitHub, GitLab, and others.

Execution of workflows Nextflow's DSL allows workflows to be deployed and run across different computing environments without having to modify the pipeline code. Nextflow comes with specific executors for various platforms, including major cloud providers. It supports the following environments for pipeline execution:

Local: This is the default executor where Nextflow pipelines run on Linux or Mac OS, and the execution occurs on the computer where the pipeline is launched. HPC workload managers: Nextflow supports workload managers such as Slurm, SGE, LSF, Moab, PBS Pro, PBS/Torque, HTCondor, NQSII, and OAR. Kubernetes: Nextflow can be used with local or cloud-based Kubernetes implementations (GKE, EKS, or AKS). Cloud batch services: It is compatible with AWS Batch and Azure Batch Other environments: Nextflow can also be used with Apache Ignite, Google Life Sciences, and various container frameworks for portability.

Containers for portability across computing environments In Nextflow, there is tight integration with software containers. Workflows and single processes can utilize containers for their execution across different computing environments, eliminating the need for complex installation and configuration routines. Nextflow supports container frameworks such as Docker, Singularity, Charliecloud, Podman, and Shifter. These containers can be automatically retrieved from external repositories when the pipeline is executed. Additionally, it was revealed at Nextflow Summit 2022 that future versions of Nextflow will support a dedicated container provisioning service for better integration of customized containers into workflows.

Developmental history Nextflow was originally developed at the Centre for Genomic Regulation in Spain and released as an open-source project on GitHub in July 2013. In October 2018, the project license for Nextflow was changed from GPLv3 to Apache 2.0. In July 2018, Seqera Labs was launched as a spin-off from the Centre for Genomic Regulation. The company employs many of Nextflow's core developers and maintainers and provides commercial services and consulting with a focus on Nextflow. In July 2020, a major extension and revision of Nextflow's domain-specific language was introduced to allow for sub-workflows and additional improvements. In the same year, monthly downloads of Nextflow reached approximately 55,000.

Adoption and reception

… excerpt ends here. Continue reading the full article.

Worked examples

Example 1 — a first encounter with Nextflow

Start with the simplest possible case. Write down what Nextflow claims or describes in one sentence, then invent the smallest concrete situation in which that sentence is true. In computer science, the smallest case is usually a single object, a single equation or a single measurement. Check that every symbol or term in your sentence has a meaning in that case.

Example 2 — changing one variable

Take the situation from Example 1 and change exactly one quantity: double it, halve it, or set it to zero. Predict what should happen to Nextflow before you calculate. Comparing your prediction with the result is the fastest way to find out whether you understand the idea or only the words.

Example 3 — an exam-style question

Typical questions about Nextflow ask you to (a) state it precisely, (b) apply it to given data, and (c) explain a limitation. Practise writing all three answers in under five minutes; the third part is what separates a full-mark answer from an average one.

Applications of Nextflow

In research: Nextflow appears in computer science research whenever the underlying quantities have to be modelled precisely. Papers usually cite it as a starting assumption and then explore where it breaks down.
In technology and industry: Engineering practice reuses Nextflow in design rules, simulations and safety margins. Knowing the idea lets you read a specification sheet and understand why the numbers look the way they do.
In the classroom: Nextflow is common in secondary-school and first-year university syllabi. It links to neighbouring topics Software using the Apache license, Workflow languages, Workflow technology, so understanding it makes those chapters shorter.
In everyday life: Look for Nextflow outside the textbook — in sport, cooking, traffic, electronics or the sky above you. An example you found yourself is remembered far longer than one you were given.

How to study Nextflow in 20 minutes

Read the reference excerpt below once, without taking notes.
Close the page and write down what Nextflow means in your own words.
Compare your version with the excerpt and mark what you missed.
Work through the three examples above with pen and paper.
Explain Nextflow out loud to somebody else — or to Teacher Smith in the lgStudy chat.

Frequently asked questions

What is Nextflow in simple terms?

Nextflow is a scientific workflow system predominantly used for bioinformatic data analysis. It establishes standards for programmatically creating a series of dependent computational steps and facilitates their execution on various local and cloud resources.

Why does Nextflow matter?

Because it connects several computer science ideas at once: it gives you a definition you can apply, a quantity you can calculate, and a way to check whether a result is plausible.

How should I study Nextflow?

Read the excerpt, restate it from memory, then work through the examples and applications listed on this page. The five-step study plan above takes about twenty minutes.

What does this page cover?

It gives you a compact reference excerpt plus original lgStudy explanations, examples, applications and study material on Nextflow.