Job Array

If you haven't done yet, download Samples by:

git clone https://hidekiCCS@bitbucket.org/hidekiCCS/hpc-workshop.git

---

If you are running a large number of serial/parallel jobs that are independent each other, it is recommended to submit them as a job array to make the best use of your allocated resources.

Use Array Task ID as a Parameter to the code

Get into JobArray1 directory under workshop,

[fuji@cypress1 ~]$ cd workshop
[fuji@cypress1 workshop]$ cd JobArray1/
[fuji@cypress1 JobArray1]$ ls
hello2.py  slurmscript

There is a python script that is

[fuji@cypress1 JobArray1]$ cat hello2.py
# HELLO PYTHON
import sys
import datetime
import socket

if (len(sys.argv) < 2):
	print '%s [taskID]' % (sys.argv[0])
        sys.exit()
#
taskID = int(sys.argv[1])
now = datetime.datetime.now()
print 'Hello, world!'
print now.isoformat()
print socket.gethostname()
print 'My Task ID = %d' % taskID

This python script takes a number from the command-line and prints it. You can run it on the login node as,

[fuji@cypress1 JobArray1]$ python ./hello2.py 123
Hello, world!
2018-08-22T13:47:28.572176
cypress1
My Task ID = 123

Look at slurmscript,

[fuji@cypress1 JobArray1]$ cat slurmscript
#!/bin/bash
#SBATCH --qos=workshop            # Quality of Service
#SBATCH --partition=workshop      # partition
#SBATCH --job-name=python       # Job Name
#SBATCH --time=00:01:00         # WallTime
#SBATCH --nodes=1               # Number of Nodes
#SBATCH --ntasks-per-node=1     # Number of tasks (MPI processes)
#SBATCH --cpus-per-task=1       # Number of threads per task (OMP threads)
#SBATCH --array=0-9             # Array of IDs=0,1,2,3,4,5,6,7,8,9

module load anaconda
python hello2.py $SLURM_ARRAY_TASK_ID

array

This

#SBATCH --array=0-9             # Array of IDs=0,1,2,3,4,5,6,7,8,9

The argument can be specific array index values, a range of index values, and an optional step size. For example,

#SBATCH --array=0-31  # Submit a job array with index values between 0 and 31

#SBATCH --array=1,3,5,7 # Submit a job array with index values of 1, 3, 5 and 7

#SBATCH --array=1-7:2 # Submit a job array with index values between 1 and 7 with a step size of 2 (i.e. 1, 3, 5 and 7)

Note that the minimum index value is zero.

Job ID and Environment Variables

Job arrays will have two additional environment variable set. SLURM_ARRAY_JOB_ID will be set to the first job ID of the array. SLURM_ARRAY_TASK_ID will be set to the job array index value. SLURM_ARRAY_TASK_COUNT will be set to the number of tasks in the job array. SLURM_ARRAY_TASK_MAX will be set to the highest job array index value. SLURM_ARRAY_TASK_MIN will be set to the lowest job array index value.

In slurmscript above, SLURM_ARRAY_TASK_ID is given to the python code as,

python hello2.py $SLURM_ARRAY_TASK_ID

so that the hello2.py prints the task ID.

Submit a Job

[fuji@cypress1 JobArray1]$ sbatch slurmscript
Submitted batch job 773958

After the job completed, you will see 10 new files as (it may take a while)

[fuji@cypress1 JobArray1]$ ls
hello2.py           slurm-773958_1.out  slurm-773958_3.out  slurm-773958_5.out  slurm-773958_7.out  slurm-773958_9.out
slurm-773958_0.out  slurm-773958_2.out  slurm-773958_4.out  slurm-773958_6.out  slurm-773958_8.out  slurmscript

The default log file name is slurm-[Job ID]_[Task_ID].out.

[fuji@cypress1 JobArray1]$ cat slurm-773958_9.out
Hello, world!
2018-08-22T14:01:44.492109
cypress01-117
My Task ID = 9

Use Array Task ID to define the script file name

Get into JobArray2 directory under workshop,

[fuji@cypress1 ~]$ cd workshop/JobArray2/
[fuji@cypress1 JobArray2]$ ls
hello2.py    script01.sh  script03.sh  script05.sh  script07.sh  script09.sh   slurmscript2
script00.sh  script02.sh  script04.sh  script06.sh  script08.sh  slurmscript1
[fuji@cypress1 JobArray2]$

Look at slurmscript1

[fuji@cypress1 JobArray2]$ cat slurmscript1
#!/bin/bash
#SBATCH --qos=workshop            # Quality of Service
#SBATCH --partition=workshop      # partition
#SBATCH --job-name=python       # Job Name
#SBATCH --time=00:01:00         # WallTime
#SBATCH --nodes=1               # Number of Nodes
#SBATCH --ntasks-per-node=1     # Number of tasks (MPI processes)
#SBATCH --cpus-per-task=1       # Number of threads per task (OMP threads)
#SBATCH --array=0-9             # Array of IDs=0,1,2,3,4,5,6,7,8,9

sh ./script$(printf "%02d" $SLURM_ARRAY_TASK_ID).sh

The last line, script$(printf "%02d" $SLURM_ARRAY_TASK_ID).sh makes the files name using $SLURM_ARRAY_TASK_ID, so it will run script00.sh, script01.sh, script00.sh, … and script09.sh.

Submit job,

[fuji@cypress1 JobArray2]$ sbatch slurmscript1
Submitted batch job 773970

Cancel Jobs in Job Array

Look at slurmscript12

[fuji@cypress1 JobArray2]$ cat slurmscript2
#!/bin/bash
#SBATCH --qos=workshop            # Quality of Service
#SBATCH --partition=workshop      # partition
#SBATCH --job-name=python       # Job Name
#SBATCH --time=00:01:00         # WallTime
#SBATCH --nodes=1               # Number of Nodes
#SBATCH --ntasks-per-node=1     # Number of tasks (MPI processes)
#SBATCH --cpus-per-task=1       # Number of threads per task (OMP threads)
#SBATCH --array=0-9             # Array of IDs=0,1,2,3,4,5,6,7,8,9

sh ./script$(printf "%02d" $SLURM_ARRAY_TASK_ID).sh

sleep 60

There is sleep 60 so each task runs 60 seconds. Submit job and look at the jobs running/queued.

[fuji@cypress1 JobArray2]$ sbatch slurmscript2
Submitted batch job 773980
[fuji@cypress1 JobArray2]$ squeue -u fuji
     JOBID    QOS               NAME     USER ST       TIME NO NODELIST(REASON)
  773980_0 worksh             python     fuji  R       0:04  1 cypress01-117
  773980_1 worksh             python     fuji  R       0:04  1 cypress01-117
  773980_2 worksh             python     fuji  R       0:04  1 cypress01-117
  773980_3 worksh             python     fuji  R       0:04  1 cypress01-117
  773980_4 worksh             python     fuji  R       0:04  1 cypress01-117
  773980_5 worksh             python     fuji  R       0:04  1 cypress01-117
  773980_6 worksh             python     fuji  R       0:04  1 cypress01-117
  773980_7 worksh             python     fuji  R       0:04  1 cypress01-117
  773980_8 worksh             python     fuji  R       0:04  1 cypress01-117
  773980_9 worksh             python     fuji  R       0:04  1 cypress01-117

To cancel 773980_1 only,

[fuji@cypress1 JobArray2]$ scancel 773980_1
[fuji@cypress1 JobArray2]$ squeue -u fuji
     JOBID    QOS               NAME     USER ST       TIME NO NODELIST(REASON)
  773980_0 worksh             python     fuji  R       0:18  1 cypress01-117
  773980_2 worksh             python     fuji  R       0:18  1 cypress01-117
  773980_3 worksh             python     fuji  R       0:18  1 cypress01-117
  773980_4 worksh             python     fuji  R       0:18  1 cypress01-117
  773980_5 worksh             python     fuji  R       0:18  1 cypress01-117
  773980_6 worksh             python     fuji  R       0:18  1 cypress01-117
  773980_7 worksh             python     fuji  R       0:18  1 cypress01-117
  773980_8 worksh             python     fuji  R       0:18  1 cypress01-117
  773980_9 worksh             python     fuji  R       0:18  1 cypress01-117

To cancel tasks 5-8,

[fuji@cypress1 JobArray2]$ scancel 773980_[5-8]
[fuji@cypress1 JobArray2]$ squeue -u fuji
     JOBID    QOS               NAME     USER ST       TIME NO NODELIST(REASON)
  773980_0 worksh             python     fuji  R       0:30  1 cypress01-117
  773980_2 worksh             python     fuji  R       0:30  1 cypress01-117
  773980_3 worksh             python     fuji  R       0:30  1 cypress01-117
  773980_4 worksh             python     fuji  R       0:30  1 cypress01-117
  773980_9 worksh             python     fuji  R       0:30  1 cypress01-117

To cancel all tasks,

[fuji@cypress1 JobArray2]$ scancel 773980
[fuji@cypress1 JobArray2]$ squeue -u fuji
     JOBID    QOS               NAME     USER ST       TIME NO NODELIST(REASON)