Overview

The Workflow Task

The Workflow task receives a ModelBuildingConfig resource id and a target task name.

The target task will trigger the execution of all its required upstream tasks, ensuring that each necessary dependency is processed in sequence until the target task is reached and executed.

For example, if cellPositionConfig is selected as the target task, Luigi will first process the CellPositionGenerator requirements, which directly depends on CellCompositionGenerator. Therefore CellCompositionGenerator will be executed before CellPositionGenerator can run.

Note

It is not necessary for the ModelBuildingConfig to include all the generator configs, however it must contain all configs up until the target task selected when executing the workflow.

Generator: Task Layout

A Generator is a Luigi Task in a predefined hierarchy of generators which has NEXUS resources as inputs and outputs.

The Generator produces an activity with the generated entity, registered to the knowledge graph. The Generated is completed if a target entity can be found in the database with the specific used_config input.

Note

Since the workflow registers and searches for resources in the knowledge graph, retriggering a task requires deprecating the corresponding activity associated with the input configuration. Without this, the target will always be found, and Luigi will consider the task as already completed, preventing re-execution.

Generator: Anatomy & Variants

Types

There are two main types of generators:

• Relay Generators

• Multi Variant Generators

Relay generators are simple tasks that propagate the input config downstream by creating an activity with a clone of the config. An example of a Relay generator is the MacroConnectomeGenerator.

Multi variant generators are the most common tasks, scattering variant tasks and then merging them to produce the final result. Each Generator that derives from MultiVariantGenerator implements a scatter and optionally a merge method.

Variants

A Generator may launch one or more variant tasks. A variant is an executable tool identified by the triplet (generator_name, variant_name, version). For more info see the variant documentation.