Changes between Version 4 and Version 5 of Workshops/JobCheckpointing/Examples


Ignore:
Timestamp:
04/01/2026 08:29:20 PM (4 months ago)
Author:
Carl Baribault
Comment:

Ongoing update with submitter/runner implementation

Legend:

Unmodified
Added
Removed
Modified

  • Workshops/JobCheckpointing/Examples

    v4 v5  
    22= Job Checkpointing Examples =
    33
    4 == Checkpoint Runner ==
    5 
    6 Here is a job script for a fully self restarting job for use with either of the accompanying, minimal working examples of checkpointed applications in BASH, Python, or R.
    7 
    8 === checkpoint_runner.sh ===
     4For a quick start see [wiki:Workshops/JobCheckpointing/Examples#WorkingExamples Working Examples] below.
     5
     6Here is an implementation for running '''sequential workflows''' on Cypress - where each component result of the workflow depends the previous component result in the workflow.
     7
     8It can be considered an implementation of an HPC workflow design pattern (see for example [https://github.com/deepakkum21/Books/blob/master/Design%20Patterns%20-%20Elements%20of%20Reusable%20Object%20Oriented%20Software%20-%20GOF.pdf Design Patterns]) that can be described by any of the following expressions.
     9
     10* submitter/runner pattern (see also [https://gjbex.github.io/Workflows-for-HPC/ Workflow for HPC])
     11* job chaining (see [https://docs.nersc.gov/jobs/best-practices/ (NERSC) Best Practices for Jobs])
     12* sequential workflows
     13* restart workflows
     14* checkpoint/restart job series
     15* using job dependencies to restart jobs that checkpoint their state”
     16
     17'''But wait''' - before you adopt this type of checkpointing workflow, consider whether your workflow can be considered
     18'''embarrassingly parallel''', where the data elements can be processed '''entirely independently''' of each other - rather than each one depending on the previous. (See also [https://www.freecodecamp.org/news/embarrassingly-parallel-algorithms-explained-with-examples/ Embarrassingly Parallel Algorithms Explained].)
     19
     20If so, namely that your case is the former (embarrassingly parallel) in the above, then please first refer to the workshop [wiki:Workshops/JobParallelism].
     21
     22== Checkpoint Submitter ==
     23
     24Here is a job script for a fully self restarting job that controls the workflow by submitting the checkpoint runner job script (see [wiki:Workshops/JobCheckpointing/Examples#CheckpointRunner Checkpoint Runner]) further below.
     25
     26=== checkpoint_submitter.sh ===
    927
    1028{{{
    1129#!/bin/bash
    1230###############################################################################
    13 # CHECKPOINT RUNNER: checkpoint + auto-requeue (Slurm 14.x-safe)
    14 ###############################################################################
    15 #   MODULE_LIST     : modules loaded before running app command
    16 #   APP_CMD         : command to run (string)
    17 #   LAUNCH_MODE     : run app command directly or via srun
    18 #   SRUN_ARGS       : arguments to srun, if needed
    19 #   TIME_LIMIT      : wall time (D-HH:MM:SS or HH:MM:SS) sync w/ #SBATCH --time
    20 #   MARGIN_SEC      : seconds before wall time to checkpoint (checkpoint_timer)
    21 #   CKPT_PATH       : path to checkpoint file
    22 #   CHECKPOINT_EVERY: checkpoint after every this many application iterations
    23 #   MAX_ITER        : stop application after this many iterations total
    24 #   MAX_RESTARTS    : stop requeuing after this many restarts (safety)
     31# CHECKPOINT SUBMITTER: self-restarting to re-submit runner
     32# Controls repeated submissions of checkpoint_runner.sh
    2533###############################################################################
    2634# User settings with defaults
     35###############################################################################
     36# CKPT_PATH            : path to checkpoint file
     37# MAX_ITER             : stop application after this many iterations total
     38# RUNNER_SCRIPT        : job script with checkpoint before end of time limit
     39# SUBMITTER_MARGIN_SEC : time margin to restart before time limit
     40# SUBMITTER_TIME_LIMIT : wall time <= 24 hours (format as --time in man sbatch)
    2741###############################################################################
    2842# Edit settings below
    2943# Or pass values via...
    30 #   sbatch --export=ALL,Var1=...,...,VarN=... checkpoint_runner.sh
     44#   sbatch --export=ALL,Var1=...,...,VarN=... checkpoint_submitter.sh
    3145# Or pass values via...
    32 #   Var1=... VarN=... sbatch checkpoint_runner.sh
    33 ###############################################################################
     46#   Var1=... VarN=... sbatch checkpoint_submitter.sh
     47###############################################################################
     48CKPT_PATH="${CKPT_PATH:-state_iter.txt}"
     49MAX_ITER="${MAX_ITER:-500}"
     50RUNNER_SCRIPT="${RUNNER_SCRIPT:-checkpoint_runner.sh}"
     51SUBMITTER_MARGIN_SEC="${SUBMITTER_MARGIN_SEC:-60}"
     52SUBMITTER_TIME_LIMIT="${SUBMITTER_TIME_LIMIT:-3:30}"
     53###############################################################################
     54# Slurm directives (keep SUBMITTER_TIME_LIMIT <= 24 hours)
     55###############################################################################
     56#SBATCH --job-name=ckpt_submitter
     57#SBATCH --output=log_submitter_%j.out
     58#SBATCH --error=log_submitter_%j.err
     59#SBATCH --time=24:00:00
     60#SBATCH --ntasks=1
     61#SBATCH --cpus-per-task=1
     62#SBATCH --mem=256M
     63#SBATCH --partition=centos7
     64#SBATCH --qos=normal
     65###############################################################################
     66# Exit on error
     67###############################################################################
     68set -euo pipefail
     69###############################################################################
     70# Helpers
     71###############################################################################
     72shopt -s expand_aliases
     73alias dtstamp='date +%Y%m%d-%H:%M:%S'
     74info(){ echo "Info[$(dtstamp)]: $*"; }
     75get_ckpt_iter() {
     76  [[ -f "${CKPT_PATH}" ]] || { echo 0; return; }
     77  tr -cd '0-9' < "${CKPT_PATH}" 2>/dev/null || echo 0
     78}
     79submit_worker() {
     80  sbatch "${RUNNER_SCRIPT}" | awk '{print $4}'
     81}
     82to_seconds() {
     83        # Acceptable time formats include "minutes", "minutes:seconds", "hours:min- utes:seconds",
     84        # "days-hours", "days-hours:minutes" and "days-hours:minutes:seconds"
     85        local input="$1"
     86        # Handle days-hours format by replacing '-' with ':' to unify the delimiter
     87        # Then split components into an array
     88        IFS=':' read -r -a parts <<< "${input//-/:}"
     89        case ${#parts[@]} in
     90            1) # Format: minutes
     91                seconds=$((parts[0] * 60))
     92                ;;
     93            2) # Formats: minutes:seconds OR days-hours
     94                if [[ $input == *"-"* ]]; then
     95                    seconds=$((parts[0] * 86400 + parts[1] * 3600))
     96                else
     97                    seconds=$((parts[0] * 60 + parts[1]))
     98                fi
     99                ;;
     100            3) # Formats: hours:minutes:seconds OR days-hours:minutes
     101                if [[ $input == *"-"* ]]; then
     102                    seconds=$((parts[0] * 86400 + parts[1] * 3600 + parts[2] * 60))
     103                else
     104                    seconds=$((parts[0] * 3600 + parts[1] * 60 + parts[2]))
     105                fi
     106                ;;
     107            4) # Format: days-hours:minutes:seconds
     108                seconds=$((parts[0] * 86400 + parts[1] * 3600 + parts[2] * 60 + parts[3]))
     109                ;;
     110            *)
     111                echo "Unsupported format"
     112                exit 1
     113                ;;
     114        esac
     115        echo "$seconds"
     116}
     117should_restart() {
     118  now=$(date +%s)
     119  elapsed=$(( now - restart_start_sec ))
     120  remaining=$(( restart_limit_sec - elapsed ))
     121  echo -ne "Time before restart: $remaining (sec)\r" >&2
     122  (( remaining < SUBMITTER_MARGIN_SEC ))
     123}
     124###############################################################################
     125# Log settings
     126###############################################################################
     127info "Start on $(hostname); JOBID=${SLURM_JOB_ID:-unknown}"
     128info "Submitter settings:"
     129info "CKPT_PATH=${CKPT_PATH}"
     130info "MAX_ITER=${MAX_ITER}"
     131info "RUNNER_SCRIPT=${RUNNER_SCRIPT}"
     132info "SUBMITTER_MARGIN_SEC=${SUBMITTER_MARGIN_SEC}"
     133info "SUBMITTER_TIME_LIMIT=${SUBMITTER_TIME_LIMIT}"
     134###############################################################################
     135# Restore state if present
     136###############################################################################
     137state_file="submitter_state.txt"
     138submitter_cycle=0
     139if [[ -f "${state_file}" ]]; then
     140  source "${state_file}"
     141  info "Loaded state: submitter_cycle=${submitter_cycle}"
     142else
     143  info "No previous state found; starting fresh."
     144fi
     145###############################################################################
     146# Track time for restart
     147###############################################################################
     148restart_start_sec=$(date +%s)
     149restart_limit_sec=$(to_seconds "${SUBMITTER_TIME_LIMIT}")
     150###############################################################################
     151# Main Loop
     152###############################################################################
     153info "Submitter cycle: ${submitter_cycle}"
     154module load anaconda3/2023.07 # using python3
     155while true; do
     156  current_iter=$(get_ckpt_iter)
     157  info "Current checkpoint iteration: ${current_iter}"
     158  if (( current_iter >= MAX_ITER )); then
     159          info "Reached MAX_ITER=${MAX_ITER}; running final analytics."
     160    # List of analytics scripts to check
     161    analytics_scripts=(
     162      "aggregate_usage.py"
     163      "plot_progress.py"
     164    )
     165          # Run resource usage aggregation
     166          if command -v python3 >/dev/null 2>&1; then
     167      # Loop over each script and run only if present
     168      for script in "${analytics_scripts[@]}"; do
     169        if [[ -f "${script}" ]]; then
     170          info "Running analytics ${script} ..."
     171          python3 "${script}"
     172        else
     173          info "WARNING: ${script} not found; skipping."
     174        fi
     175      done
     176          else
     177            info "WARNING: python3 not found; skipping analytics."
     178          fi
     179          info "Workflow complete. Exiting submitter."
     180    exit 0
     181
     182  fi
     183  JOBID=$(submit_worker)
     184  info "Submitted worker: ${JOBID}"
     185        # Log worker JobIDs for later aggregation
     186        id_log="worker_ids.txt"
     187        echo "${JOBID}" >> "${id_log}"
     188  # Wait for worker to finish
     189  info "Waiting on first of worker ${JOBID} to finish or self-restart."
     190  while squeue -j "${JOBID}" >/dev/null 2>&1; do
     191    sleep 10
     192    if should_restart; then
     193      info "Submitter approaching timeout -> restarting self"
     194      # Save state
     195      echo "submitter_cycle=$((submitter_cycle+1))" > "${state_file}"
     196      # Submit a new submitter instance
     197      sbatch checkpoint_submitter.sh
     198      exit 0
     199
     200    fi
     201  done
     202  # Read worker exit code
     203  rc=$(sacct -j "${JOBID}" --format=ExitCode -P | awk -F: 'NR==2{print $1}')
     204  info "Worker finished with code ${rc}"
     205  case "${rc}" in
     206    0)
     207      info "Worker reports completed workflow."
     208      exit 0
     209
     210      ;;
     211    99|124)
     212      info "Checkpoint advance -> continuing workflow."
     213      ;;
     214    *)
     215      info "Unexpected worker failure (code ${rc}). Stopping."
     216      exit "${rc}"
     217
     218      ;;
     219  esac
     220  # Loop continues to submit next worker
     221done
     222}}}
     223
     224== Checkpoint Runner ==
     225
     226Here is the checkpoint runner job script for use with the following.
     227
     228* for jobs submitted by [wiki:Workshops/JobCheckpointing/Examples#CheckpointSubmitter Checkpoint Submitter] above
     229* for running any of the following, minimal working examples of checkpointed applications in BASH, Python, or R.
     230 * For BASH, see [wiki:Workshops/JobCheckpointing/Examples#BASHCheckpointingExample BASH Checkpointint Example]
     231 * For Python, see [wiki:Workshops/JobCheckpointing/Examples#PythonCheckpointingExample Python Checkpointint Example]
     232 * For R, see [wiki:Workshops/JobCheckpointing/Examples#RCheckpointingExample R Checkpointint Example]
     233
     234=== checkpoint_runner.sh ===
     235
     236{{{
     237#!/bin/bash
     238###############################################################################
     239# CHECKPOINT RUNNER: run application w/ checkpoint + controlled stop
     240# Use with checkpoint_submitter.sh to re-submit as needed
     241###############################################################################
     242# User settings with defaults
     243###############################################################################
     244# APP_CMD          : command to run (string) - must handle signal SIGTERM
     245# CHECKPOINT_EVERY : checkpoint after every this many application iterations
     246# CKPT_PATH        : path to checkpoint file
     247# LAUNCH_MODE      : run app command directly or via srun
     248# MAX_ITER         : stop application after this many iterations total
     249# MODULE_LIST      : modules loaded before running app command
     250# RUNNER_MARGIN_SEC: seconds before wall time to checkpoint (for timeout)
     251# RUNNER_TIME_LIMIT: wall time <= 24 hours (format as --time in man sbatch)
     252# SRUN_ARGS        : arguments to srun, if needed
     253###############################################################################
     254# Edit settings below
     255# Or pass values via...
     256#   sbatch --export=ALL,Var1=...,...,VarN=... checkpoint_submitter.sh
     257# Or pass values via...
     258#   Var1=... VarN=... sbatch checkpoint_submitter.sh
     259###############################################################################
     260APP_CMD="${APP_CMD:-python3 checkpoint_signal_iter.py}"
     261CHECKPOINT_EVERY="${CHECKPOINT_EVERY:-20}"
     262CKPT_PATH="${CKPT_PATH:-state_iter.txt}"
     263LAUNCH_MODE="${LAUNCH_MODE:-direct}"
     264MAX_ITER="${MAX_ITER:-500}"
    34265MODULE_LIST="${MODULE_LIST:-anaconda3/2023.07}"
    35                                            # space-delimited module list
    36 APP_CMD="${APP_CMD:-python3 checkpoint_signal_iter.py}"
    37 LAUNCH_MODE="${LAUNCH_MODE:-direct}"       # direct | srun
    38 SRUN_ARGS="${SRUN_ARGS:--n 1}"             # extra srun flags
    39 TIME_LIMIT="${TIME_LIMIT:-00:03:00}"       # match #SBATCH --time below
    40 MARGIN_SEC="${MARGIN_SEC:-60}"             # checkpoint time before time limit
    41 CKPT_PATH="${CKPT_PATH:-state_iter.txt}"   # checkpoint file path
    42 CHECKPOINT_EVERY="${CHECKPOINT_EVERY:-20}" # number of iter. before checkpoint
    43 MAX_ITER="${MAX_ITER:-500}"                # number of iter. total
    44 MAX_RESTARTS="${MAX_RESTARTS:-10}"         # max. number of restarts
    45 
    46 ###############################################################################
    47 # Slurm directives (keep TIME_LIMIT in sync with #SBATCH --time)
    48 ###############################################################################
    49 #SBATCH --job-name=ckpt_requeue_demo
    50 #SBATCH --output=log_%j.out
    51 #SBATCH --error=log_%j.err
    52 #SBATCH --open-mode=append
    53 #SBATCH --time=00:03:00                 # keep in sync with TIME_LIMIT above
     266RUNNER_MARGIN_SEC="${RUNNER_MARGIN_SEC:-60}"
     267RUNNER_TIME_LIMIT="${RUNNER_TIME_LIMIT:-00:03:00}"
     268SRUN_ARGS="${SRUN_ARGS:--n 1}"
     269###############################################################################
     270# Slurm directives (keep RUNNER_TIME_LIMIT <= 24 hours)
     271###############################################################################
     272#SBATCH --job-name=ckpt_runner_demo
     273#SBATCH --output=log_runner_%j.out
     274#SBATCH --error=log_runner_%j.err
     275#SBATCH --time=00:03:00
    54276#SBATCH --ntasks=1
    55277#SBATCH --cpus-per-task=1
    56278#SBATCH --mem=512M
    57 #SBATCH --requeue
    58 #SBATCH --get-user-env
    59 #SBATCH --partition=centos7             # change partition to defq if needed
    60 #SBATCH --qos=normal                    # best qos for node availability
    61 
     279#SBATCH --partition=centos7
     280#SBATCH --qos=normal
     281###############################################################################
     282# Exit on error
     283###############################################################################
    62284set -euo pipefail
     285###############################################################################
     286# Helpers
     287###############################################################################
    63288shopt -s expand_aliases
    64289alias dtstamp="date +%Y%m%d-%H:%M:%S"
    65290info(){ echo "Info[$(dtstamp)]: $*"; }
    66 
    67 info "Start on $(hostname); JOB_ID=${SLURM_JOB_ID}; RESTARTS=${SLURM_RESTART_COUNT:-0}"
    68 info "Settings:"
     291get_ckpt_iter() {
     292  [[ -f "${CKPT_PATH}" ]] || { echo 0; return; }
     293  tr -cd '0-9' < "${CKPT_PATH}" 2>/dev/null || echo 0
     294}
     295to_seconds() {
     296        # Acceptable time formats include "minutes", "minutes:seconds", "hours:min- utes:seconds",
     297        # "days-hours", "days-hours:minutes" and "days-hours:minutes:seconds"
     298        local input="$1"
     299        # Handle days-hours format by replacing '-' with ':' to unify the delimiter
     300        # Then split components into an array
     301        IFS=':' read -r -a parts <<< "${input//-/:}"
     302        case ${#parts[@]} in
     303            1) # Format: minutes
     304                seconds=$((parts[0] * 60))
     305                ;;
     306            2) # Formats: minutes:seconds OR days-hours
     307                if [[ $input == *"-"* ]]; then
     308                    seconds=$((parts[0] * 86400 + parts[1] * 3600))
     309                else
     310                    seconds=$((parts[0] * 60 + parts[1]))
     311                fi
     312                ;;
     313            3) # Formats: hours:minutes:seconds OR days-hours:minutes
     314                if [[ $input == *"-"* ]]; then
     315                    seconds=$((parts[0] * 86400 + parts[1] * 3600 + parts[2] * 60))
     316                else
     317                    seconds=$((parts[0] * 3600 + parts[1] * 60 + parts[2]))
     318                fi
     319                ;;
     320            4) # Format: days-hours:minutes:seconds
     321                seconds=$((parts[0] * 86400 + parts[1] * 3600 + parts[2] * 60 + parts[3]))
     322                ;;
     323            *)
     324                echo "Unsupported format"
     325                exit 1
     326                ;;
     327        esac
     328        echo "$seconds"
     329}
     330###############################################################################
     331# Log settings
     332###############################################################################
     333info "Start on $(hostname); JOBID=${SLURM_JOB_ID:-unknown}"
     334info "Runner settings:"
     335info "APP_CMD=${APP_CMD}"
     336info "CHECKPOINT_EVERY=${CHECKPOINT_EVERY}"
     337info "CKPT_PATH=${CKPT_PATH}"
     338info "LAUNCH_MODE=${LAUNCH_MODE}"
     339info "MAX_ITER=${MAX_ITER}"
    69340info "MODULE_LIST=${MODULE_LIST}"
    70 info "APP_CMD=${APP_CMD}"
    71 info "LAUNCH_MODE=${LAUNCH_MODE}"
     341info "RUNNER_MARGIN_SEC=${RUNNER_MARGIN_SEC}"
     342info "RUNNER_TIME_LIMIT=${RUNNER_TIME_LIMIT}"
    72343info "SRUN_ARGS=${SRUN_ARGS}"
    73 info "TIME_LIMIT=${TIME_LIMIT}"
    74 info "MARGIN_SEC=${MARGIN_SEC}"
    75 info "CKPT_PATH=${CKPT_PATH}"
    76 info "CHECKPOINT_EVERY=${CHECKPOINT_EVERY}"
    77 info "MAX_ITER=${MAX_ITER}"
    78 info "MAX_RESTARTS=${MAX_RESTARTS}"
    79 
    80 # Load site modules (available on Cypress workers)
    81 module load slurm 2>/dev/null || true   # makes scontrol visible on worker
    82 module load "${MODULE_LIST}" || true
    83 
    84 # Tool paths
    85 SCTRL="$(command -v scontrol || true)"
    86 
    87 # Short diagnostics
    88 if [[ -n "${SCTRL}" ]]; then
    89   echo "=== BEGIN JOB SNAPSHOT (scontrol) ==="
    90   "${SCTRL}" show job "${SLURM_JOB_ID}" | \
    91     grep -E "JobId=|Partition=|QOS=|TimeLimit=|StartTime=|EndTime=|RunTime=|State=|Restarts="
    92   echo "=== END JOB SNAPSHOT (scontrol) ==="
    93 else
    94   info "WARNING: scontrol not found on this node; in-place requeue will not be attempted."
    95 fi
    96 
    97 # Helpers
    98 get_ckpt_iter() {
    99   # reads the first numeric token from CKPT_PATH; returns 0 on any error
    100   [[ -f "${CKPT_PATH}" ]] || { echo 0; return; }
    101   local v
    102   v=$(tr -cd '0-9' < "${CKPT_PATH}" 2>/dev/null)
    103   echo "${v:-0}"
    104 }
    105 
    106 get_restarts() {
    107   if [[ -n "${SLURM_RESTART_COUNT:-}" ]]; then
    108     echo "${SLURM_RESTART_COUNT}"
    109   elif [[ -n "${SCTRL}" ]]; then
    110     "${SCTRL}" show job "${SLURM_JOB_ID}" | tr ' ' '\n' | awk -F= '/^Restarts=/{print $2; exit}'
    111   else
    112     echo "0"
    113   fi
    114 }
    115 
    116 to_seconds() {
    117   local t="$1"
    118   if [[ "$t" == *-*:*:* ]]; then
    119     local d h m s; IFS='-:' read -r d h m s <<<"$t"
    120     echo $(( d*86400 + h*3600 + m*60 + s ))
    121   else
    122     local h m s; IFS=':' read -r h m s <<<"$t"
    123     h=${h:-0}; m=${m:-0}; s=${s:-0}
    124     echo $(( h*3600 + m*60 + s ))
    125   fi
    126 }
    127 
    128 # Trap (batch shell)
    129 signal_handler () {
    130   info "TERM/INT caught in batch shell"
    131   local rc_local=0
    132 
    133   if [[ -n "${child_pid:-}" ]]; then
    134     # srun or bash are group leaders; forward to their process group
    135     local child_pgid
    136     child_pgid="$(ps -o pgid= -p "${child_pid}" 2>/dev/null | awk '{print $1}')"
    137     if [[ -n "${child_pgid}" ]]; then
    138       kill -TERM "-${child_pgid}" 2>/dev/null || true
    139     else
    140       kill -TERM "${child_pid}" 2>/dev/null || true
    141     fi
    142     wait "${child_pid}" || rc_local=$?
    143   fi
    144 
    145   info "Program exit code (from trap): ${rc_local}"
    146   local restarts; restarts=$(get_restarts)
    147   if [[ ${rc_local} -eq 99 && ${requeued:-0} -eq 0 && ${restarts} -lt ${MAX_RESTARTS} && -n "${SCTRL}" ]]; then
    148     requeued=1
    149     info "Checkpoint written (trap path). Requeueing in-place (same JobID)..."
    150     "${SCTRL}" requeue "${SLURM_JOB_ID}" || true
    151     info "Requeued via scontrol."
    152   fi
    153   exit 0
    154 }
    155 trap 'signal_handler' TERM INT
    156 
    157 # Launch via timeout
    158 TOTAL_SEC=$(to_seconds "${TIME_LIMIT}")
    159 RUN_WINDOW_SEC=$(( TOTAL_SEC - MARGIN_SEC ))
     344###############################################################################
     345# Load requested modules
     346###############################################################################
     347info "Loading modules."
     348module load "${MODULE_LIST}"
     349###############################################################################
     350# Timeout window
     351###############################################################################
     352TOTAL_SEC=$(to_seconds "${RUNNER_TIME_LIMIT}")
     353RUN_WINDOW_SEC=$(( TOTAL_SEC - RUNNER_MARGIN_SEC ))
    160354if (( RUN_WINDOW_SEC <= 0 )); then
    161355  info "WARNING: RUN_WINDOW_SEC <= 0; using 1s."
    162356  RUN_WINDOW_SEC=1
    163357fi
    164 
    165358before_iter=$(get_ckpt_iter)
    166 
    167359set +e
    168360if [[ "${LAUNCH_MODE}" == "srun" ]]; then
    … …  
    176368rc=$?
    177369set -e
    178 
    179370info "Program exit code (from timeout wrapper): ${rc}"
    180 
    181 # Interpret coreutils 8.4 returns:
    182 #   0   -> completed
    183 #   99  -> app exited 99 before timeout expiry (valid)
    184 #   124 -> timeout expired (TERM sent), treat as checkpoint cycle and requeue
    185 #          (optionally confirm CKPT grew)
    186 #   else-> unexpected -> propagate
    187 requeued=0
     371###############################################################################
     372# Interpret exit-code from timeout
     373#   0   = completed all iterations (no further submissions necessary)
     374#   99  = app voluntarily checkpointed early -> submitter should continue
     375#   124 = timeout before walltime expiry -> checkpoint likely written -> continue
     376###############################################################################
    188377if [[ ${rc} -eq 0 ]]; then
    189   info "Completed."
     378  info "Application finished all iterations."
    190379  exit 0
    191380elif [[ ${rc} -eq 99 ]]; then
    192   restarts=$(get_restarts)
    193   if (( restarts < MAX_RESTARTS )) && [[ -n "${SCTRL}" ]]; then
    194     requeued=1
    195     info "Checkpoint written. Requeueing in-place (same JobID)..."
    196     "${SCTRL}" requeue "${SLURM_JOB_ID}" || true
    197     info "Requeued via scontrol."
     381  info "Application exited after writing checkpoint."
     382  exit 99
     383elif [[ ${rc} -eq 124 ]]; then
     384  after_iter=$(get_ckpt_iter)
     385  if (( after_iter > before_iter )); then
     386    info "Timeout + checkpoint advance detected (${before_iter}->${after_iter})."
     387    exit 124
    198388  else
    199     info "WARNING: cannot requeue (scontrol unavailable or MAX_RESTARTS reached)."
     389    info "Timeout but checkpoint did not advance. Marking failure."
     390    exit 1
    200391  fi
    201   exit 0
    202 elif [[ ${rc} -eq 124 ]]; then
    203         after_iter=$(get_ckpt_iter)
    204         if (( after_iter > before_iter )); then
    205           restarts=$(get_restarts)
    206           if (( restarts < MAX_RESTARTS )) && [[ -n "${SCTRL}" ]]; then
    207             requeued=1
    208             info "Timeout TERM observed; checkpoint advanced (${before_iter}->${after_iter}). Requeueing..."
    209             "${SCTRL}" requeue "${SLURM_JOB_ID}" || true
    210             info "Requeued via scontrol."
    211             exit 0
    212           else
    213             info "WARNING: cannot requeue (scontrol unavailable or MAX_RESTARTS reached)."
    214             exit 0
    215           fi
    216         else
    217           info "Timeout TERM observed but checkpoint did not advance; marking as failure."
    218           exit 1
    219         fi
    220392else
    221393  info "Unexpected exit code: ${rc}"
    … …  
    224396}}}
    225397
    226 == BASH Checkpointing Example ==
     398== Working Examples ==
     399
     400=== BASH Checkpointing Example ===
    227401
    228402See [wiki:Workshops/JobCheckpointing/Examples/BASH BASH Checkpointing Example]
    229403
    230 == Python Checkpointing Example ==
     404=== Python Checkpointing Example ===
    231405
    232406See [wiki:Workshops/JobCheckpointing/Examples/Python Python Checkpointing Example]
    233407
    234 == R Checkpointing Example ==
     408=== R Checkpointing Example ===
    235409
    236410See [wiki:Workshops/JobCheckpointing/Examples/R R Checkpointing Example]