= Job Dependency =
If you haven't done yet, download Samples by:

{{{git clone https://hidekiCCS:@bitbucket.org/hidekiCCS/hpc-workshop.git}}}

----

Job dependencies are used to defer the start of a job until the specified dependencies have been satisfied. They are specified with the '''--dependency''' option to '''sbatch''' command.
{{{
sbatch --dependency=<type:job_id[:job_id][,type:job_id[:job_id]]> ...
}}}

Dependency types:
* '''after:jobid[:jobid...]'''	job can begin after the specified jobs have started
* '''afterany:jobid[:jobid...]'''	job can begin after the specified jobs have terminated
* '''afternotok:jobid[:jobid...]'''	job can begin after the specified jobs have failed
* '''afterok:jobid[:jobid...]'''	job can begin after the specified jobs have run to completion with an exit code of zero (see the user guide for caveats).


== Submitting Dependent Jobs ==
Get into '''!JobDependencies''' directory under '''workshop''', 
{{{
[fuji@cypress1 ~]$ cd workshop/
[fuji@cypress1 workshop]$ cd JobDependencies
[fuji@cypress1 JobDependencies]$ ls
addOne.py  number.dat  script.sh  slurmscript  SubmitDependentJobs.sh
}}}

Python code addOne.py reads '''number.dat''' and gets an integer number, and then adds one and stores it back to '''number.dat''' .
{{{
[fuji@cypress1 JobDependencies]$ cat addOne.py
# HELLO PYTHON
import datetime
import socket

now = datetime.datetime.now()
print 'Hello, world!'
print now.isoformat()
print socket.gethostname()
#
with open('number.dat','r') as f:
	data = f.readline()
	number = int(data)
#
print "Number = %d" % number
with open('number.dat','w') as f:
	f.write(str(number + 1))
#
}}}


'''slurmscipt''' just run the code,
{{{
[fuji@cypress1 JobDependencies]$ cat slurmscript
#!/bin/bash
#SBATCH --qos=workshop            # Quality of Service
#SBATCH --partition=workshop      # partition
#SBATCH --job-name=python       # Job Name
#SBATCH --time=00:01:00         # WallTime
#SBATCH --nodes=1               # Number of Nodes
#SBATCH --ntasks-per-node=1     # Number of tasks (MPI processes)
#SBATCH --cpus-per-task=1       # Number of threads per task (OMP threads)

module load anaconda
python addOne.py

sleep 10
}}}

Let submit one job and then submit anther job that depends on the first jobs as,
{{{
[fuji@cypress1 JobDependencies]$ sbatch slurmscript
Submitted batch job 773997
[fuji@cypress1 JobDependencies]$ sbatch --dependency=afterok:773997 slurmscript
Submitted batch job 773998
}}}

List the jobs,
{{{
[fuji@cypress1 JobDependencies]$ squeue -u fuji
     JOBID    QOS               NAME     USER ST       TIME NO NODELIST(REASON)
    773998 worksh             python     fuji PD       0:00  1 (Dependency)
    773997 worksh             python     fuji  R       0:00  1 cypress01-117
}}}

After the first job completed, the second job begin to run,
{{{
[fuji@cypress1 JobDependencies]$ squeue -u fuji
     JOBID    QOS               NAME     USER ST       TIME NO NODELIST(REASON)
    773998 worksh             python     fuji  R       0:05  1 cypress01-117
}}}

The results are:
{{{
[fuji@cypress1 JobDependencies]$ ls
addOne.py  number.dat  script.sh  slurm-773997.out  slurm-773998.out  slurmscript  SubmitDependentJobs.sh
[fuji@cypress1 JobDependencies]$ cat slurm-773997.out
Hello, world!
2018-08-22T14:55:37.421310
cypress01-117
Number = 2
[fuji@cypress1 JobDependencies]$ cat slurm-773998.out
Hello, world!
2018-08-22T14:55:47.619183
cypress01-117
Number = 3
}}}

== Submitting Many Dependent Jobs with Bash Script ==

Look at '''!SubmitDependentJobs.sh'''
{{{
[fuji@cypress1 JobDependencies]$ cat SubmitDependentJobs.sh
#!/bin/bash
EMAIL=$USER@tulane.edu
WALLTIME_LIMIT=1:00:00
export WORKDIR=`pwd`
#
QUEUE='--partition=workshop --qos=workshop'
WALLTIME="--time=$WALLTIME_LIMIT"
RESORCE="--nodes=1 --ntasks-per-node=1 --cpus-per-task=1"
OTHERS="--export=ALL --mail-type=END --mail-user=$EMAIL"
#
JOB_SETTING="$QUEUE $WALLTIME $RESORCE $OTHERS"

DEPENDENCY=""

while [[ $# > 0 ]]
do
	JOB=`sbatch --job-name=$1 $DEPENDENCY $JOB_SETTING ./$1 | awk '{print $4}'`;
	echo $JOB submitted;
	DEPENDENCY="--dependency=afterok:$JOB" ;
	shift
done
}}}

This bash script takes script names as command-line options, and submits a sequence of dependent jobs with those scripts.

The bash script, '''script.sh''' 
{{{
[fuji@cypress1 JobDependencies]$ cat script.sh
#!/bin/bash

module load anaconda
python addOne.py

sleep 1
}}}
just runs '''addOne.py'''.

Let's submit 10 of '''script.sh''',
{{{
[fuji@cypress1 JobDependencies]$ ./SubmitDependentJobs.sh script.sh script.sh script.sh script.sh script.sh script.sh script.sh script.sh script.sh script.sh
774001 submitted
774002 submitted
774003 submitted
774004 submitted
774005 submitted
774006 submitted
774007 submitted
774008 submitted
774009 submitted
774010 submitted
}}}

List jobs,
{{{
[fuji@cypress1 JobDependencies]$ squeue -u fuji
     JOBID    QOS               NAME     USER ST       TIME NO NODELIST(REASON)
    774005 worksh          script.sh     fuji PD       0:00  1 (Dependency)
    774006 worksh          script.sh     fuji PD       0:00  1 (Dependency)
    774007 worksh          script.sh     fuji PD       0:00  1 (Dependency)
    774008 worksh          script.sh     fuji PD       0:00  1 (Dependency)
    774009 worksh          script.sh     fuji PD       0:00  1 (Dependency)
    774010 worksh          script.sh     fuji PD       0:00  1 (Dependency)
    774004 worksh          script.sh     fuji  R       0:01  1 cypress01-117
}}}