Changes between Version 1 and Version 2 of Workshops/cypress/JobDependency

Timestamp:

08/22/18 15:12:46 (7 years ago)

Author:

fuji

Comment:

—

Workshops/cypress/JobDependency

@@ -111 +111 @@
 Number = 3
 }}}
+
+== Submitting Many Dependent Jobs with Bash Script ==
+
+Look at '''SubmitDependentJobs.sh'''
+{{{
+[fuji@cypress1 JobDependencies]$ cat SubmitDependentJobs.sh
+#!/bin/bash
+EMAIL=$USER@tulane.edu
+WALLTIME_LIMIT=1:00:00
+export WORKDIR=`pwd`
+#
+QUEUE='--partition=workshop --qos=workshop'
+WALLTIME="--time=$WALLTIME_LIMIT"
+RESORCE="--nodes=1 --ntasks-per-node=1 --cpus-per-task=1"
+OTHERS="--export=ALL --mail-type=END --mail-user=$EMAIL"
+#
+JOB_SETTING="$QUEUE $WALLTIME $RESORCE $OTHERS"
+
+DEPENDENCY=""
+
+while [[ $# > 0 ]]
+do
+        JOB=`sbatch --job-name=$DIRNAME$1 $DEPENDENCY $JOB_SETTING ./$1 | awk '{print $4}'`;
+        echo $JOB submitted;
+        DEPENDENCY="--dependency=afterok:$JOB" ;
+        shift
+done
+}}}
+
+This bash script takes script names as command-line options, and submits a sequence of dependent jobs with those scripts.
+
+The bash script, '''script.sh''' is
+{{{
+[fuji@cypress1 JobDependencies]$ cat script.sh
+#!/bin/bash
+
+module load anaconda
+python addOne.py
+
+sleep 1
+}}}
+runs '''addOne.py'''.
+
+Let's submit 10 of '''script.sh''',
+{{{
+[fuji@cypress1 JobDependencies]$ ./SubmitDependentJobs.sh script.sh script.sh script.sh script.sh script.sh script.sh script.sh script.sh script.sh script.sh
+774001 submitted
+774002 submitted
+774003 submitted
+774004 submitted
+774005 submitted
+774006 submitted
+774007 submitted
+774008 submitted
+774009 submitted
+774010 submitted
+}}}
+
+List jobs,
+{{{
+[fuji@cypress1 JobDependencies]$ squeue -u fuji
+     JOBID    QOS               NAME     USER ST       TIME NO NODELIST(REASON)
+    774005 worksh          script.sh     fuji PD       0:00  1 (Dependency)
+    774006 worksh          script.sh     fuji PD       0:00  1 (Dependency)
+    774007 worksh          script.sh     fuji PD       0:00  1 (Dependency)
+    774008 worksh          script.sh     fuji PD       0:00  1 (Dependency)
+    774009 worksh          script.sh     fuji PD       0:00  1 (Dependency)
+    774010 worksh          script.sh     fuji PD       0:00  1 (Dependency)
+    774004 worksh          script.sh     fuji  R       0:01  1 cypress01-117
+}}}