bbp_workflow.task

Collection of base tasks for the Workflow engine.

Classes

EnvCfg(*args, **kwargs)	Standard set of environment configuration parameters.
IPyParallel(*args, **kwargs)	Allocate ipyparallel cluster and run remote_script.
IPyParallelExclusive(*args, **kwargs)	IPyParallel which uses exclusive nodes with engine(task) per cpus.
KgCfg(*args, **kwargs)	Knowledge Graph configuration parameters.
KgTask(*args, **kwargs)	Base class for Knowledge Graph enabled tasks.
LookupKgEntity(*args, **kwargs)	Base class for Knowledge Graph lookup tasks.
MakeRemoteFolder(*args, **kwargs)	Creates a folder on the host file system.
MakeVirtualEnv(*args, **kwargs)	Creates a python virtual environment on the host file system.
OutputCfg(*args, **kwargs)	Common configuration for tasks producing output at the specified file system location.
RemoteHostCfg(*args, **kwargs)	Add host parameter to your task using this mixin.
SbatchTask(*args, **kwargs)	Sbatch task.
SlurmCfg(*args, **kwargs)	Standard set of Slurm configuration parameters.
SrunTask(*args, **kwargs)	Srun task.

class bbp_workflow.task.EnvCfg(*args, **kwargs)

Bases: Config

Standard set of environment configuration parameters.

enable_internet

env

module_archive

module_path

modules

virtual_env

class bbp_workflow.task.IPyParallel(*args, **kwargs)

Bases: _JobPathMixin, Task

Allocate ipyparallel cluster and run remote_script.

Inherits SlurmCfg, EnvCfg parameters, enabling customization of this task.

args

modules

ntasks

remote_script()

Will run in the context of ipyparallel cluster.

Use Client to talk to the cluster:

def remote_script(self):
    from ipyparallel import Client
    client = Client()
    lview = client.load_balanced_view()
    lview.block = True
    def f(_):
        return 'foo'
    result = lview.map(f, range(2))
    print(result)

class bbp_workflow.task.IPyParallelExclusive(*args, **kwargs)

Bases: IPyParallel

IPyParallel which uses exclusive nodes with engine(task) per cpus.

cpus_per_task

exclusive

mem

class bbp_workflow.task.KgCfg(*args, **kwargs)

Bases: Config

Knowledge Graph configuration parameters.

kg_base

kg_org

kg_proj

class bbp_workflow.task.KgTask(*args, **kwargs)

Bases: KgCfg, Task

Base class for Knowledge Graph enabled tasks.

done(msg, entity)

Mark output as done and print entity URL.

from_file(*args, **kwargs)

Delegate to DataDownload from_file method using kg params from the task.

from_id(entity_cls, resource_id, on_no_result=None)

Delegate to entity from_id method using kg params from the task.

from_json_str(*args, **kwargs)

Delegate to DataDownload from_json_str method using kg params from the task.

make_web_link(resource_id)

Make link to nexus web from the resource_id.

name

output()

RunAnywayTarget.

publish(entity, **kwargs)

Delegate to entity publish method using kg params from the task.

class bbp_workflow.task.LookupKgEntity(*args, **kwargs)

Bases: KgCfg, ExternalTask

Base class for Knowledge Graph lookup tasks.

url

class bbp_workflow.task.MakeRemoteFolder(*args, **kwargs)

Bases: Task

Creates a folder on the host file system.

output()

RemoteTarget (path=self.path).

path

class bbp_workflow.task.MakeVirtualEnv(*args, **kwargs)

Bases: Task

Creates a python virtual environment on the host file system.

Usage:

bbp-workflow launch --follow bbp_workflow.task MakeVirtualEnv \
    virtual-env=/gpfs/bbp.cscs.ch/home/${USER}/tmp/venv \
    modules=py-bbp-workflow \
    packages='bmtk'

install_luigi

module_archive

modules

output()

RemoteTarget(self.host, self.virtual_env).

packages

requirements

virtual_env

class bbp_workflow.task.OutputCfg(*args, **kwargs)

Bases: Config

Common configuration for tasks producing output at the specified file system location.

path_prefix

class bbp_workflow.task.RemoteHostCfg(*args, **kwargs)

Bases: Config

Add host parameter to your task using this mixin.

host

class bbp_workflow.task.SbatchTask(*args, **kwargs)

Bases: _JobPathMixin, Task

Sbatch task.

Inherits SlurmCfg, EnvCfg, RemoteHostCfg parameters, enabling customization of this task.

args

command

class bbp_workflow.task.SlurmCfg(*args, **kwargs)

Bases: Config

Standard set of Slurm configuration parameters.

Slurm based tasks will extend this class, so they can be parametrized the same way.

account

chdir

constraint

cpus_per_task

exclusive

job_name

job_output

mem

mem_per_cpu

mpi

nodes

ntasks

ntasks_per_node

partition

qos

time

class bbp_workflow.task.SrunTask(*args, **kwargs)

Bases: Task

Srun task.

Inherits SlurmCfg, EnvCfg, RemoteHostCfg parameters, enabling customization of this task.

args

command

return_result

run()

Will run self.command on the self.host using SLURM allocation.

If the task which extends SrunTask implements self.remote_script, the source of the self.remote_script method will be stored in a tmp file. This tmp file will be submitted as the first argument to the self.command and self.args will follow.