GNU Parallel

GNU parallel is a shell tool for executing many small independent tasks on the multi-core platform (compute nodes). A job can be a single command or a small script that has to be run for each of the lines in the input. GNU parallel can then split the input and pipe it into commands in parallel.

Adding GNU Parallel to Environment

To use GNU parallel on the command line,

module load parallel

GNU Parallel Syntax

• Reading commands to be run in parallel from an input file

    parallel [OPTIONS] < CMDFILE
  

• Reading command arguments on the command line

    parallel [OPTIONS] COMMAND [ARGUMENTS] ::: ARGLIST
  

• Reading command arguments from an input file

    parallel [OPTIONS] COMMAND [ARGUMENTS] :::: ARGFILE
  

​https://www.gnu.org/software/parallel/parallel_tutorial.html

Many Task Computing

For example, you have many scripts to run,

[fuji@cypress1 JobArray2]$ ls
hello2.py    script01.sh  script03.sh  script05.sh  script07.sh  script09.sh   slurmscript2
script00.sh  script02.sh  script04.sh  script06.sh  script08.sh  slurmscript1

To run all distributed over 4 cores,

[fuji@cypress1 JobArray2]$ parallel -j 4 sh ::: script??.sh