brun_fastsurfer.sh

Usage

./brun_fastsurfer.sh --help
Script to run FastSurfer on multiple subjects in parallel/series.

Usage:
brun_fastsurfer.sh --subjects_list <subjects_list_path> [other options]
OR
brun_fastsurfer.sh --subjects <subject_id>=<t1_path> [<subject_id>=<t1_path>
    [...]] [other options]
OR
brun_fastsurfer.sh [other options]

Other options:
brun_fastsurfer.sh [...] [--batch "<i>/<n>"] [--parallel <n>|max]
    [--parallel_seg <n>|max] [--parallel_surf <n>|max]
    [--run_fastsurfer <command/run_fastsurfer_script>]
    [--statusfile <filename>] [--debug] [--help]
    [<fastsurfer_flags>]

License:  Apache License, Version 2.0

Documentation of Options:
Generally, brun_fastsurfer works similar to run_fastsurfer, but loops over
multiple subjects from
i. a list passed through stdin of the format (one subject per line)
---
<subject_id>=<t1_path>[ <subject_specific_parameters>[ ...]]
...
---
ii. a subjects list file using the same format (use Ctrl-D to end the input), or
iii. a list of subjects directly passed (this does not support subject-specific
  parameters)

A path or parameter that contains a space has to be quoted or escaped as it
would be in the shell, i.e. '/data/my subject/t1.mgz', "/data/my subject/t1.mgz"
or /data/my\ subject/t1.mgz. Single quotes are the simplest, because everything
inside them is taken literally, including backslashes. No expansion is performed
in either kind of quotes, so a $ or a ` is just that character.

--batch "<i>/<n>": run the i-th of n batches (starting at 1) of the full list of
  subjects (default: 1/1, == run all). "slurm_task_id" is a valid option for
  "<i>".
  Note, brun_fastsurfer.sh will also automatically detect being run in a SLURM
  JOBARRAY and split according to $SLURM_ARRAY_TASK_ID and
  $SLURM_ARRAY_TASK_COUNT (unless values are specifically assigned with the
  --batch argument).
--parallel <n>|max: parallel execution of run_fastsurfer for <n> images. Creates
  <n> processes with each process performing segmentation and surface
  reconstruction. The default is this serial execution mode with n=1:
  '--parallel 1'.
--parallel_seg <n>|max and
--parallel_surf <m>|max: activate independent segmentation and surface
  reconstruction pipelines. Segmentation and Surface reconstruction have
  independent processing queues. After successful segmentation (<n> parallel
  processes), cases are transferred into the surface queue (<m> parallel
  processes). Together max. m+n processes will run. Logfiles unchanged, console
  output for individual subjects is interleaved with subject_id prepended.
--run_fastsurfer <command>: This option enables the startup of fastsurfer in a
  more controlled manner, for example to delegate the fastsurfer run to
  container:
  --run_fastsurfer "singularity exec --nv --no-mount home,cwd -e -B <host_dir>:/data /fastsurfer/run_fastsurfer.sh"
  Note, paths to files and --sd have to be defined in the container file system
  in this case.
--statusfile <filename>: a file to document which subject ran successfully. Also
  used to skip surface recon, if the previous segmentation failed.
--threads <n>,
--threads_seg <n>, and
--threads_surf <n>: specify number of threads for each parallel "process", i.e.
  total_threads=num_seg_processes * num_seg_threads + num_surf_processes *
  num_surf_threads.
--debug: Additional debug output.
--help: print this help.

With the exception of --t1 and --sid, all run_fastsurfer.sh options are
supported, see 'run_fastsurfer.sh --help'.

This tool requires functions in stools.sh (expected in same folder as this
script).

Subjects Lists

The input files and options may be specified in three ways:

  1. By writing them into the console (or by piping them in) (default) (one case per line),

  2. by passing a subjects list file --subjects_list <subjects_list_path> (one case per line), or

  3. by passing them on the command line --subjects "<subject_id>=<t1_path>" [more cases] (no additional options supported).

These files/input options will usually be in the format <subject_id>=<t1_path> [additional options], where additional options are optional and enable passing options different to the “general options” given on the command line to brun_fastsurfer.sh. One example for such a case-specific option is an optional T2w image (e.g. for the HypVINN). An example subjects list file might look like this:

001=/home/user/my_mri_data/001/t1_weighted.nii.gz --t2 /home/user/my_mri_data/001/t2_weighted.nii.gz
002=/home/user/my_mri_data/002/t1_weighted.nii.gz --t2 /home/user/my_mri_data/002/t2_weighted.nii.gz
003=/home/user/my_mri_data/003/t1_weighted_alt.nii.gz --t2 /home/user/my_mri_data/003/t2_weighted.nii.gz
...

Parallelization with brun_fastsurfer.sh

brun_fastsurfer.sh has powerful builtin parallel processing capabilities. These are hidden underneath the --parallel* <n>|max and the --device <torch_device> as well as --viewagg_device <torch_device> flags. One of the core properties of FastSurfer is the split into the segmentation (which uses Deep Learning and therefore benefits from GPUs) and the surface pipeline (which does not benefit from GPUs). For ideal batch processing, we want different resource scheduling.

--parallel* allows three parallel batch processing modes: serial, single parallel pipeline and dual parallel pipeline.

Serial processing (default)

Each case/image is processed after the other fully, i.e. surface reconstruction of case 1 is fully finished before segmentation of case 2 is started. This setting is the default and represents the manual flags --parallel 1.

Single parallel pipeline

This mode is ideal for CPU-based processing for segmentation. It will process segmentations and surfaces in series in the same process, but multiple cases are processed at the same time.

export FASTSURFER_HOME=${FASTSURFER_HOME:-/path/to/FastSurfer}
$FASTSURFER_HOME/brun_fastsurfer.sh --parallel 4 --threads 2

will start 4 segmentations (and surface reconstructions) at the same time, and will start a fifth, when the surface processing of one of the four first cases is finished (--parallel 4). It will try to use 2 threads per case (--threads 2) and perform reconstruction of left and right hemispheres in parallel (--threads 2, 2 >= 2). --parallel max will remove the limit and start all cases at the same time (each with the target number of threads given by --threads).

Without --threads (and without an exported OMP_NUM_THREADS), each case would choose its threads itself and take what auto gives the whole machine. So when more than one case runs at a time, brun_fastsurfer.sh divides that between them instead and passes each case its share as --threads_seg and --threads_surf, within the same limits auto keeps to (at most 4 threads for a segmentation with --device cuda or mps, 8 otherwise). This also reaches cases started in a container through --run_fastsurfer. A subject line that sets its own --threads keeps it, on top of the shares of the others, and brun_fastsurfer.sh names such lines in a warning.

Dual parallel pipeline

This is ideal for GPU-based processing for segmentation. It will process segmentations and surfaces in separate pipelines, which is useful for optimized GPU loading. Multiple cases may be processed at the same time.

export FASTSURFER_HOME=${FASTSURFER_HOME:-/path/to/FastSurfer}
$FASTSURFER_HOME/brun_fastsurfer.sh --device cuda:0-1 --parallel_seg 2 \
  --parallel_surf max --threads_seg 8 --threads_surf 4

will start 2 parallel segmentations (--parallel_seg 2) using GPU 0 for case 1 and GPU 1 for case 2 (--device cuda:0-1 – same as --device cuda:0,1). After one of these segmentations is finished, the segmentation of case 3 will start on that same device as well as the surface reconstruction (without putting a limit on parallel surface reconstructions, --parallel_surf max). Each segmentation process will aim to use 8 threads/cores (--threads_seg 8) and each surface reconstruction process will aim to use 4 threads (--threads_surf 4) with both hemispheres processed in parallel (--threads_surf 4, 4 >= 2, so right hemisphere will use 2 threads and left as well).

Note, if your GPU has sufficient video memory, two parallel segmentations can run on the same GPU, but the script cannot schedule more than one process per GPU for multiple GPUs, i.e. --device cuda:0,1 --parallel_seg 4 is not supported.

Questions

Can I disable the progress bars in the output?

You can disable the progress bars by setting the TQDM_DISABLE environment variable to 1, if you have tqdm>=4.66.

For docker, this can be done with the flag -e, e.g. docker run -e TQDM_DISABLE=1 ..., for singularity with the flag --env, e.g. singularity exec --env TQDM_DISABLE=1 ... and for native installations by prepending, e.g. TQDM_DISABLE=1 $FASTSURFER_HOME/run_fastsurfer.sh ....