Skip to content

Parallelization

These options control parallel processing (e.g., multiple subjects at once),

n_jobs module-attribute

Python
n_jobs: int = 1

Specifies how many subjects you want to process in parallel. If 1, disables parallel processing.

parallel_backend module-attribute

Python
parallel_backend: Literal['loky', 'dask'] = 'loky'

Specifies which backend to use for parallel job execution. loky is the default backend used by joblib. dask requires Dask to be installed. Ignored if n_jobs is set to 1.

dask_open_dashboard module-attribute

Python
dask_open_dashboard: bool = False

Whether to open the Dask dashboard in the default webbrowser automatically. Ignored if parallel_backend is not 'dask'.

dask_temp_dir module-attribute

Python
dask_temp_dir: PathLike | None = None

The temporary directory to use by Dask. Dask places lock-files in this directory, and also uses it to "spill" RAM contents to disk if the amount of free memory in the system hits a critical low. It is recommended to point this to a location on a fast, local disk (i.e., not a network-attached storage) to ensure good performance. The directory needs to be writable and will be created if it does not exist.

If None, will use .dask-worker-space inside of deriv_root.

dask_worker_memory_limit module-attribute

Python
dask_worker_memory_limit: str = '10G'

The maximum amount of RAM per Dask worker.

dask_worker_startup_timeout module-attribute

Python
dask_worker_startup_timeout: float = 3600

How long to wait (in seconds) for workers to come up before giving up, e.g. while dask_cluster jobs sit in the batch queue. Only used when dask_cluster is set.

dask_cluster module-attribute

Python
dask_cluster: str | Callable[[], Any] | None = None

Attach to an external Dask cluster instead of starting workers locally, e.g., on an HPC system via dask-jobqueue. Either the address of a running scheduler (e.g., "tcp://10.0.0.1:8786"), or a callable that returns a Dask cluster object (or anything accepted by distributed.Client):

Python
def dask_cluster():
    from dask_jobqueue import SLURMCluster

    cluster = SLURMCluster(queue="cpu", cores=8, memory="32GB", walltime="02:00:00")
    cluster.scale(jobs=4)  # or cluster.adapt(maximum_jobs=8)
    return cluster

If the callable returns a cluster with no workers requested, the pipeline calls cluster.scale(n_jobs). Compute nodes must share the filesystem holding bids_root and deriv_root and have the same Python environment. Ignored if parallel_backend is not 'dask'; dask_worker_memory_limit and dask_temp_dir are then also ignored (configure these on the cluster instead).