bbp_workflow.simulation.task¶

Collection of simulation tasks for the Workflow engine.

Classes

`AmendSimulationCampaign`(args, *kwargs)	Generate additional simulations in the campaign with extended filter function.
`CampaignSimProcessorSbatch`(args, *kwargs)	Simulation from the campaign processor base task.
`CleanUpSimCampaignFiles`(args, *kwargs)	Clean up files from the simulation campaign folders.
`CortexNrdmsPySim`(args, *kwargs)	Cortex simulation task.
`GenerateAndRunSimCampaign`(args, *kwargs)	Generate and run simulation campaign.
`GenerateCoupledCoordsAndRunSimCampaign`(...)	Generate and run coupled coords simulation campaign.
`GenerateCoupledCoordsSimulationCampaign`(...)	Generate simulation campaign with coupled coords values.
`GenerateSimulationCampaign`(args, *kwargs)	Generate simulation campaign.
`HippocampusNrdmsPySim`(args, *kwargs)	Hippocampus simulation task.
`LookupDetailedCircuit`(args, *kwargs)	Lookup detailed circuit knowledge graph entity.
`LookupSimCampaign`(args, *kwargs)	Alias for `LookupSimulationCampaign`.
`LookupSimulationCampaign`(args, *kwargs)	Lookup simulation campaign configuration Knowledge Graph entity.
`MoveSimCampaign`(args, *kwargs)	Move simulation campaign configuration to the new path prefix.
`MultiscaleNrdmsPySim`(args, *kwargs)	Multiscale simulation task.
`NrdmsPySim`(args, *kwargs)	Base neurodamus-py simulation task.
`PlasticityCortexNrdmsPySim`(args, *kwargs)	Cortex plasticity simulation task.
`RegisterCoupledCoordsSimulationCampaign`(...)	Register an existing simulation campaign with coupled coords.
`RegisterSimCampaignFromConfig`(args, *kwargs)	Register simulation campaign configuration from the existing config.json.
`RegisterSimulationCampaign`(args, *kwargs)	Register an existing simulation campaign.
`SimCampaign`(args, *kwargs)	Run simulation campaign.
`SimCampaignHalt`(args, *kwargs)	Stop simulation campaign.
`SimCampaignInfo`(args, *kwargs)	Print simulation campaign configuration.
`SimCampaignMixin`()	Shared helper methods for old and new sim campaign tasks.
`SimCampaignThrottle`(args, *kwargs)	Change the amount of simulations running in parallel for the campaign.
`SimulationCampaign`(args, *kwargs)	Run simulation campaign.
`SimulationCampaignProcessor`(args, *kwargs)	Apply processor tasks to individual simulations form the campaign.
`ThalamusNrdmsPySim`(args, *kwargs)	Thalamus simulation task.

class bbp_workflow.simulation.task.AmendSimulationCampaign(*args, **kwargs)¶

Bases: KgTask

Generate additional simulations in the campaign with extended filter function.

Usage:

simulation.cfg¶

 [DEFAULT]
 module-archive: archive/2022-02

 [GenerateSimulationCampaign]
 coords: {"ca": [1.3, 1.35],
          "stim_number_mcs": [40, 60]}
 attrs: {"path_prefix": "/gpfs/bbp.cscs.ch/path/where/to/generate",
         "blue_config_template": "BlueConfig.tmpl",
         "user_target": "user.target",
         "mg": 1.0,
         "depolarization": 100.0,
         "target": "MiniColumn_0",
         "duration": 2500,
         "seed": 123456}
 meta-seed: 123
 coords_filter_func: my_module.filter_function
 param-processors: ["bbp_workflow.sci.default_depolarization_treatment",
                    "bbp_workflow.sci.default_ca_treatment"]

 [AmendSimulationCampaign]
 coords_filter_func: my_module.filter_function_ext

bbp-workflow launch --follow --config simulation.cfg \
    bbp_workflow.simulation AmendSimulationCampaign \
        sim-config-url=https://bbp.epfl.ch/nexus/v1/resources/bbp/project/_/sim-config

coords_filter_func = Filter function to select simulations.(None)¶

requires()¶: LookupSimulationCampaign.

sim_config_url = Optionally provide simulation campaign URL or configure the URL directly on the LookupSimulationCampaign task.(None)¶

class bbp_workflow.simulation.task.CampaignSimProcessorSbatch(*args, **kwargs)¶

Bases: SbatchTask

Simulation from the campaign processor base task.

coords = Simulation conditions.¶

is_sonata = Set to true in order to use "simulation_config.json".(False)¶

requires()¶: Require simulation.

class bbp_workflow.simulation.task.CleanUpSimCampaignFiles(*args, **kwargs)¶

Bases: Task

Clean up files from the simulation campaign folders.

Usage:

bbp-workflow launch --follow \
    bbp_workflow.simulation CleanUpSimCampaignFiles \
        sim-config-url=https://bbp.epfl.ch/nexus/v1/resources/bbp/project/_/sim-cfg-id \
        files='["AllCompartments.bbp"]'

files = List of files for deletion from each individual simulation folder.¶

requires()¶: LookupSimulationCampaign.

sim_config_url = Optionally provide simulation campaign URL or configure the URL directly on the LookupSimulationCampaign task.(None)¶

class bbp_workflow.simulation.task.CortexNrdmsPySim(*args, **kwargs)¶

Bases: NrdmsPySim

Cortex simulation task.

Run simulation defined by the simulation config located in the self.chdir folder.

Usage:

bbp-workflow launch --follow \
    bbp_workflow.simulation CortexNrdmsPySim \
        account=proj123 \
        chdir=/gpfs/bbp.cscs.ch/path/to/blue/config/folder

container = Singularity container to use.(neurodamus-neocortex__1.13-2.16.6-2.8.1-amd64.sif)¶

modules = Environment modules to load.(neurodamus-neocortex py-neurodamus)¶

class bbp_workflow.simulation.task.GenerateAndRunSimCampaign(*args, **kwargs)¶

Bases: SimCampaign

Generate and run simulation campaign.

circuit_url = Optionally provide detailed circuit URL or configure the URL directly on the LookupDetailedCircuit task.(None)¶

requires()¶: GenerateSimulationCampaign.

class bbp_workflow.simulation.task.GenerateCoupledCoordsAndRunSimCampaign(*args, **kwargs)¶

Bases: GenerateAndRunSimCampaign

Generate and run coupled coords simulation campaign.

requires()¶: GenerateCoupledCoordsSimulationCampaign.

class bbp_workflow.simulation.task.GenerateCoupledCoordsSimulationCampaign(*args, **kwargs)¶

Bases: GenerateSimulationCampaign

Generate simulation campaign with coupled coords values.

coords = Coordinates of the scanned values. Generated scan will contain zip'ed tuples of the coords values across all the dimensions. Make sure coords are of equal length across all the dimensions.({})¶

class bbp_workflow.simulation.task.GenerateSimulationCampaign(*args, **kwargs)¶

Bases: KgTask

Generate simulation campaign.

Produce simulation configuration files in unique folders at path_prefix gpfs location. Simulation configuration files are generated by replacing any blue_config_template placeholder with values from coords, attrs or any additional mapping that param-processors return. The amount of simulation configurations is determined by the Cartesian product of coords values.

As a result configuration entity will be created in the Knowledge Graph with generation activity linking to the detailed circuit.

Usage:

simulation.cfg¶

 [GenerateSimulationCampaign]
 coords: {"ca": [1.3, 1.35],
          "stim_number_mcs": [40, 60]}
 attrs: {"path_prefix": "/gpfs/bbp.cscs.ch/path/where/to/generate",
         "blue_config_template": "BlueConfig.tmpl",
         "user_target": "user.target",
         "mg": 1.0,
         "depolarization": 100.0,
         "target": "MiniColumn_0",
         "duration": 2500,
         "seed": 123456}
 coords_filter_func: my_module.filter_function
 param-processors: ["bbp_workflow.sci.default_depolarization_treatment",
                    "bbp_workflow.sci.default_ca_treatment"]

bbp-workflow launch --follow --config simulation.cfg \
    bbp_workflow.simulation GenerateSimulationCampaign \
        circuit-url=https://bbp.epfl.ch/nexus/v1/resources/bbp/project/_/circuit-id

attrs = Additional static attributes of the campaign. Required to have at least `path_prefix` and `blue_config_template`. Optionally can contain `user_target` file path which will be copied to the simulation folders and `user_target_file_name` placeholder will contain the destination path which can be used in the template. Optional `rnd_seed_range` list with two arguments to numpy randint function will generate seed value for the campaign and make `seed` placeholder available in the template.¶

circuit_url = Detailed circuit url to use.¶

coords = Coordinates of the scanned values. Generated scan will contain Cartesian product of the coords values across all the dimensions.({})¶

coords_filter_func = Filter function to select simulations.(None)¶

description = Description of the simulation campaign.(None)¶

max_workers = Parallelization level of campaign generation. Default value will be taken from the ``workers`` parameter in the [DEFAULT] section of the cfg file.(0)¶

meta_seed = Seed for the generator of `seed` placeholder random values.(None)¶

modules = Environment modules to load for post-processors.(py-bbp-workflow)¶

name = Name of the simulation campaign.(None)¶

parallel_jobs = Parallelization level of post-processors. Default value will be taken from the ``workers`` parameter in the [DEFAULT] section of the cfg file.(0)¶

param_processors = List of parameter processing functions.([])¶

post_processors = List of post-processing functions.([])¶

requires()¶: LookupDetailedCircuit (url=self.circuit_url).

seed_as_coord = Provide dictionary(e.g.{"low":0,"high":10,"size":4}) in order to update `coords` with additional `seed` dimension of specified `size` and unique random values from `low`~`high` range.({})¶

validate_output = Set to False if you do not want output file to be validate with bluepy-configfile.(True)¶

class bbp_workflow.simulation.task.HippocampusNrdmsPySim(*args, **kwargs)¶

Bases: NrdmsPySim

Hippocampus simulation task.

Run simulation defined by the simulation config located in the self.chdir folder.

Usage:

bbp-workflow launch --follow \
    bbp_workflow.simulation HippocampusNrdmsPySim \
        account=proj123 \
        chdir=/gpfs/bbp.cscs.ch/path/to/blue/config/folder

container = Singularity container to use.(neurodamus-hippocampus__1.8-2.16.6-2.8.1-amd64.sif)¶

modules = Environment modules to load.(neurodamus-hippocampus py-neurodamus)¶

class bbp_workflow.simulation.task.LookupDetailedCircuit(*args, **kwargs)¶

Bases: LookupKgEntity

Lookup detailed circuit knowledge graph entity.

output()¶: KgUrlTarget( DetailedCircuit, self.url).

class bbp_workflow.simulation.task.LookupSimCampaign(*args, **kwargs)¶

Bases: LookupSimulationCampaign

Alias for LookupSimulationCampaign.

class bbp_workflow.simulation.task.LookupSimulationCampaign(*args, **kwargs)¶

Bases: LookupKgEntity

Lookup simulation campaign configuration Knowledge Graph entity.

output()¶: KgUrlTarget( SimCampaignConf, self.url).

class bbp_workflow.simulation.task.MoveSimCampaign(*args, **kwargs)¶

Bases: Task

Move simulation campaign configuration to the new path prefix.

Usage:

bbp-workflow launch --follow \
    bbp_workflow.simulation MoveSimCampaign \
        sim-config-url=https://bbp.epfl.ch/nexus/v1/resources/bbp/project/_/sim-cfg-id \
        path-prefix=/gpfs/bbp.cscs.ch/path/where/to/move

path_prefix = New GPFS path prefix where to move all simulation campaign files.¶

requires()¶: LookupSimulationCampaign.

sim_config_url = Optionally provide simulation campaign URL or configure the URL directly on the LookupSimulationCampaign task.(None)¶

class bbp_workflow.simulation.task.MultiscaleNrdmsPySim(*args, **kwargs)¶

Bases: NrdmsPySim

Multiscale simulation task.

Run simulation defined by the simulation config located in the self.chdir folder.

Usage:

bbp-workflow launch --follow \
    bbp_workflow.simulation MultiscaleNrdmsPySim \
        account=proj123 \
        chdir=/gpfs/bbp.cscs.ch/path/to/blue/config/folder

command = The command `sbatch` will schedule for execution.(multiscale-run compute)¶

cpus_per_task = Amount of cpus allocated per task.(0)¶

modules = Environment modules to load.(py-multiscale-run)¶

ntasks_per_node = Number of tasks to launch on each allocated node.(32)¶

class bbp_workflow.simulation.task.NrdmsPySim(*args, **kwargs)¶

Bases: SbatchTask

Base neurodamus-py simulation task.

Run simulation defined by the BlueConfig located in the self.chdir folder. For specific circuit simulation please use ThalamusNrdmsPySim, CortexNrdmsPySim or HippocampusNrdmsPySim.

command = The command `sbatch` will schedule for execution.(None)¶

container = Singularity container to use.(None)¶

cpus_per_task = Amount of cpus allocated per task.(2)¶

enable_shm = Enables the use of /dev/shm for coreneuron_input.(None)¶

exclusive = Allocate nodes exclusively and do not share them with other jobs.(True)¶

is_sonata = Set to true in order to use "simulation_config.json".(False)¶

mem = Real memory required per node.(0)¶

model_building_steps = Model building steps. Default value of 0 means: not used.(0)¶

output()¶: RemoteTarget(path = {self.chdir}/BlueConfig.SUCCESS) or RemoteTarget(path = {self.chdir}/simulation_config.json.SUCCESS).

class bbp_workflow.simulation.task.PlasticityCortexNrdmsPySim(*args, **kwargs)¶

Bases: CortexNrdmsPySim

Cortex plasticity simulation task.

Run simulation defined by the simulation config located in the self.chdir folder.

Usage:

bbp-workflow launch --follow \
    bbp_workflow.simulation PlasticityCortexNrdmsPySim \
        account=proj123 \
        chdir=/gpfs/bbp.cscs.ch/path/to/blue/config/folder

modules = Environment modules to load.(neurodamus-neocortex-plasticity py-neurodamus)¶

class bbp_workflow.simulation.task.RegisterCoupledCoordsSimulationCampaign(*args, **kwargs)¶

Bases: RegisterSimulationCampaign

Register an existing simulation campaign with coupled coords.

class bbp_workflow.simulation.task.RegisterSimCampaignFromConfig(*args, **kwargs)¶

Bases: KgTask

Register simulation campaign configuration from the existing config.json.

Usage:

bbp-workflow launch --follow \
    bbp_workflow.simulation RegisterSimCampaignFromConfig \
        name="Sim campaign name" \
        description="Sim campaign description" \
        kg-proj=my-nexus-project \
        config-path=/gpfs/bbp.cscs.ch/path/config.json

config_path = GPFS path to the `config.json`.¶

description = (None)¶

name = (None)¶

class bbp_workflow.simulation.task.RegisterSimulationCampaign(*args, **kwargs)¶

Bases: KgTask

Register an existing simulation campaign.

Parameters:

folder_name (luigi.Parameter) – Folder in the path_prefix where the simulation campaign files are located.
paths (luigi.ListParameter) – Nested list of paths corresponding to the dimensions in self.coords.
coords (luigi.DictParameter) – Coordinates of the scanned values.
attrs (luigi.DictParameter) – Additional static attributes of the campaign.
name (luigi.OptionalParameter) – Name of the simulation campaign.
description (luigi.OptionalParameter) – Description of the simulation campaign.

class bbp_workflow.simulation.task.SimCampaign(*args, **kwargs)¶

Bases: SimCampaignMixin, Task

Run simulation campaign.

Example below for large amount(100) of small simulations(10 processes per sim).

Usage:

simulation.cfg¶

 [DEFAULT]
 kg-proj: project
 account: proj123

 [SimulationCampaign]
 exclusive: False
 mem:
 cpus-per-task: 0
 ntasks: 10
 time: 30:00
 simulation-type: CortexNrdmsPySim

bbp-workflow launch --follow --config simulation.cfg \
    bbp_workflow.simulation SimCampaign \
        sim-config-url=https://bbp.epfl.ch/nexus/v1/resources/bbp/project/_/sim-cfg-id \
        parallel-jobs=100

benchmark = If True, runs last simulation from the campaign and allows to verify if allocated resources are enough for completion.(False)¶

enable_shm = Enables the use of /dev/shm for coreneuron_input.(None)¶

output()¶: [RemoteTarget(path = cfg.SUCCESS)].

requires()¶: LookupSimulationCampaign.

sim_config_url = Optionally provide simulation campaign URL or configure the URL directly on the LookupSimulationCampaign task.(None)¶

sim_indices = Indices of the simulations to run. For example: 0-10,20(None)¶

class bbp_workflow.simulation.task.SimCampaignHalt(*args, **kwargs)¶

Bases: Task

Stop simulation campaign.

Usage:

bbp-workflow launch --follow --config simulation.cfg \
    bbp_workflow.simulation SimCampaignHalt \
        sim-config-url=https://bbp.epfl.ch/nexus/v1/resources/bbp/project/_/sim-cfg-id

force = By default halts only pending jobs from the array, already queued/running will finish. Set to True in order to scancel all.(False)¶

requires()¶: LookupSimulationCampaign.

sim_config_url = Optionally provide simulation campaign URL or configure the URL directly on the LookupSimulationCampaign task.(None)¶

class bbp_workflow.simulation.task.SimCampaignInfo(*args, **kwargs)¶

Bases: Task

Print simulation campaign configuration.

Usage:

bbp-workflow launch --follow \
    bbp_workflow.simulation SimCampaignInfo \
        sim-config-url=https://bbp.epfl.ch/nexus/v1/resources/bbp/project/_/sim-cfg-id

generate_symlinks = Provide this param in order to generate human-readable folder structure with symlinks for the simulation campaign.(False)¶

requires()¶: LookupSimulationCampaign.

sim_config_url = Optionally provide simulation campaign URL or configure the URL directly on the LookupSimulationCampaign task.(None)¶

class bbp_workflow.simulation.task.SimCampaignMixin¶

Bases: object

Shared helper methods for old and new sim campaign tasks.

parallel_jobs = Amount of simulations running in parallel.(8)¶

class bbp_workflow.simulation.task.SimCampaignThrottle(*args, **kwargs)¶

Bases: Task

Change the amount of simulations running in parallel for the campaign.

Usage:

bbp-workflow launch --follow --config simulation.cfg \
    bbp_workflow.simulation SimCampaignThrottle \
        sim-config-url=https://bbp.epfl.ch/nexus/v1/resources/bbp/project/_/sim-cfg-id \
        parallel-jobs=20

job_id = If multiple jobs are running, be specific about which job should be affected.(None)¶

parallel_jobs = Amount of simulations running in parallel.(10)¶

requires()¶: LookupSimulationCampaign.

sim_config_url = Optionally provide simulation campaign URL or configure the URL directly on the LookupSimulationCampaign task.(None)¶

class bbp_workflow.simulation.task.SimulationCampaign(*args, **kwargs)¶

Bases: WrapperTask

Run simulation campaign.

Inherits params from SlurmCfg which are going to be propagated to the individual simulations of the campaign.

Usage:

simulation.cfg¶

 [DEFAULT]
 kg-proj: project
 account: proj123

 [SimulationCampaign]
 nodes: 10
 time: 3:00:00
 model-building-steps: 2
 simulation-type: CortexNrdmsPySim

bbp-workflow launch --follow --config simulation.cfg \
    bbp_workflow.simulation SimulationCampaign \
        sim-config-url=https://bbp.epfl.ch/nexus/v1/resources/bbp/project/_/sim-cfg-id

benchmark = If True, runs last simulation from the campaign and allows to verify if allocated resources are enough for completion.(False)¶

cpus_per_task = Amount of cpus allocated per task.(2)¶

enable_shm = Enables the use of /dev/shm for coreneuron_input.(None)¶

exclusive = Allocate simulation nodes exclusively.(True)¶

mem = Real memory required per node.(0)¶

model_building_steps = Optionally set the number of ModelBuildingSteps for the CoreNeuron simulation.(0)¶

requires()¶: List of self.simulation_type.

sim_config_url = Optionally provide simulation campaign URL or configure the URL directly on the LookupSimulationCampaign task.(None)¶

simulation_type = Specific simulation task class. For example: CortexNrdmsPySim, ThalamusNrdmsPySim, HippocampusNrdmsPySim.¶

class bbp_workflow.simulation.task.SimulationCampaignProcessor(*args, **kwargs)¶

Bases: WrapperTask

Apply processor tasks to individual simulations form the campaign.

coords_filter = Selects subset of coords.({})¶

processor_task = Custom task applied to each simulation.¶

requires()¶: Require simulation campaign individual processor tasks.

sim_config_url = Optionally provide simulation campaign URL or configure the URL directly on the LookupSimulationCampaign task.(None)¶

class bbp_workflow.simulation.task.ThalamusNrdmsPySim(*args, **kwargs)¶

Bases: NrdmsPySim

Thalamus simulation task.

Run simulation defined by the simulation config located in the self.chdir folder.

Usage:

bbp-workflow launch --follow \
    bbp_workflow.simulation ThalamusNrdmsPySim \
        account=proj123 \
        chdir=/gpfs/bbp.cscs.ch/path/to/blue/config/folder

modules = Environment modules to load.(neurodamus-thalamus py-neurodamus)¶