Changes between Version 1 and Version 2 of Workshops/cypress/JobDependency


Ignore:
Timestamp:
Aug 22, 2018 3:12:46 PM (3 years ago)
Author:
fuji
Comment:

Legend:

Unmodified
Added
Removed
Modified
  • Workshops/cypress/JobDependency

    v1 v2  
    111111Number = 3
    112112}}}
     113
     114== Submitting Many Dependent Jobs with Bash Script ==
     115
     116Look at '''SubmitDependentJobs.sh'''
     117{{{
     118[fuji@cypress1 JobDependencies]$ cat SubmitDependentJobs.sh
     119#!/bin/bash
     120EMAIL=$USER@tulane.edu
     121WALLTIME_LIMIT=1:00:00
     122export WORKDIR=`pwd`
     123#
     124QUEUE='--partition=workshop --qos=workshop'
     125WALLTIME="--time=$WALLTIME_LIMIT"
     126RESORCE="--nodes=1 --ntasks-per-node=1 --cpus-per-task=1"
     127OTHERS="--export=ALL --mail-type=END --mail-user=$EMAIL"
     128#
     129JOB_SETTING="$QUEUE $WALLTIME $RESORCE $OTHERS"
     130
     131DEPENDENCY=""
     132
     133while [[ $# > 0 ]]
     134do
     135        JOB=`sbatch --job-name=$DIRNAME$1 $DEPENDENCY $JOB_SETTING ./$1 | awk '{print $4}'`;
     136        echo $JOB submitted;
     137        DEPENDENCY="--dependency=afterok:$JOB" ;
     138        shift
     139done
     140}}}
     141
     142This bash script takes script names as command-line options, and submits a sequence of dependent jobs with those scripts.
     143
     144The bash script, '''script.sh''' is
     145{{{
     146[fuji@cypress1 JobDependencies]$ cat script.sh
     147#!/bin/bash
     148
     149module load anaconda
     150python addOne.py
     151
     152sleep 1
     153}}}
     154runs '''addOne.py'''.
     155
     156Let's submit 10 of '''script.sh''',
     157{{{
     158[fuji@cypress1 JobDependencies]$ ./SubmitDependentJobs.sh script.sh script.sh script.sh script.sh script.sh script.sh script.sh script.sh script.sh script.sh
     159774001 submitted
     160774002 submitted
     161774003 submitted
     162774004 submitted
     163774005 submitted
     164774006 submitted
     165774007 submitted
     166774008 submitted
     167774009 submitted
     168774010 submitted
     169}}}
     170
     171List jobs,
     172{{{
     173[fuji@cypress1 JobDependencies]$ squeue -u fuji
     174     JOBID    QOS               NAME     USER ST       TIME NO NODELIST(REASON)
     175    774005 worksh          script.sh     fuji PD       0:00  1 (Dependency)
     176    774006 worksh          script.sh     fuji PD       0:00  1 (Dependency)
     177    774007 worksh          script.sh     fuji PD       0:00  1 (Dependency)
     178    774008 worksh          script.sh     fuji PD       0:00  1 (Dependency)
     179    774009 worksh          script.sh     fuji PD       0:00  1 (Dependency)
     180    774010 worksh          script.sh     fuji PD       0:00  1 (Dependency)
     181    774004 worksh          script.sh     fuji  R       0:01  1 cypress01-117
     182}}}