BASH Checkpointing Example

BASH Checkpointing Application

Here is an example of a checkpointing application implemented in BASH script.

To run the example, see Running the BASH checkpointing example on Cypress below.

checkpoint_signal_iter.sh

#!/usr/bin/env bash
# checkpoint_signal_iter.sh
set -euo pipefail

CKPT="${CKPT_PATH:-state_iter.txt}"
EVERY="${CHECKPOINT_EVERY:-20}"
MAX_ITER="${MAX_ITER:-500}"
i=0

save() {
  # atomic write: tmp + mv
  local val="$1"
  local dir; dir="$(dirname "$(readlink -f "$CKPT")")"
  mkdir -p "$dir"
  local tmp; tmp="$(mktemp -p "$dir" .ckpt.XXXXXX)"
  printf '%s\n' "$val" > "$tmp"
  # Optional (broad) flush for older coreutils: uncomment if you want it
  # sync   # (no -d on coreutils 8.4)
  mv -f "$tmp" "$CKPT"
}

# --- load checkpoint if present ---
if [[ -f "$CKPT" ]]; then
  if [[ "$(cat "$CKPT" 2>/dev/null | tr -d '\r\n' | sed -e 's/[^0-9]//g')" != "" ]]; then
    i="$(cat "$CKPT")"
  fi
fi

# --- handle TERM: save current i and exit(99) to trigger requeue ---
term_handler() {
  echo "Bash SIGTERM: saving i=${i} and exiting 99"
  save "$i"
  exit 99
}
trap 'term_handler' TERM

echo "Resuming from i=${i} (every ${EVERY}, MAX_ITER=${MAX_ITER})"
while true; do
  i=$(( i + 1 ))
  sleep 1

  if (( i % EVERY == 0 )); then
    save "$i"
    echo "[periodic/iter] saved i=${i}"
  fi

  if (( i > MAX_ITER )); then
    echo "Reached i=${i} > ${MAX_ITER}; exiting 0"
    save "$i"
    exit 0
  fi
done

Running the BASH checkpointing example on Cypress

To run the BASH checkpointing job example, defaulting to checkpointing every 20 application iterations and a total of 500 iterations, perform the following.

1. Create your own versions of the following files in your current directory on Cypress. (For file editing with nano on Cypress, see ​File Editing Example.)

• checkpoint_submitter.sh
• checkpoint_runner.sh
• checkpoint_signal_iter.sh

2. Change permissions on the BASH application script, checkpoint_signal_iter.sh executable via the following command.

[tulaneID@cypress1 ~]$ chmod u+x checkpoint_signal_iter.sh

3. Submit the job via the following command.

[tulaneID@cypress1 ~]$ APP_CMD="./checkpoint_signal_iter.sh" \
CKPT_PATH=state_iter_bash.txt \
sbatch checkpoint_runner.sh

4. You can monitor the jobs' outputs as they are updated by repeatedly executing the following command, using Ctrl+C to exit files as they finish updating.

[tulaneID@cypress1 ~]$ tail -f log_*

Sample run outputs

List of output files

Here is a list of all output files from a sample run sorted by time, latest-to-earliest

[tulaneID@cypress2 ~]$ls -lt *.csv *.err *.out *.png *.txt
-rw-r--r-- 1 tulaneID groupID   452 Apr  1 17:57 anomaly_details.csv
-rw-r--r-- 1 tulaneID groupID 26376 Apr  1 17:57 anomaly_stats_table.png
-rw-r--r-- 1 tulaneID groupID  1402 Apr  1 17:57 log_submitter_3303154.out
-rw-r--r-- 1 tulaneID groupID 40571 Apr  1 17:57 resource_trend.png
-rw-r--r-- 1 tulaneID groupID 52107 Apr  1 17:57 progress_with_anomalies.png
-rw-r--r-- 1 tulaneID groupID   495 Apr  1 17:57 worker_usage.csv
-rw-r--r-- 1 tulaneID groupID   461 Apr  1 17:57 log_submitter_3303150.err
-rw-r--r-- 1 tulaneID groupID   761 Apr  1 17:57 log_submitter_3303150.out
-rw-r--r-- 1 tulaneID groupID     0 Apr  1 17:57 log_submitter_3303154.err
-rw-r--r-- 1 tulaneID groupID    18 Apr  1 17:57 submitter_state.txt
-rw-r--r-- 1 tulaneID groupID   818 Apr  1 17:54 log_runner_3303151.out
-rw------- 1 tulaneID groupID     4 Apr  1 17:54 state_iter_bash.txt
-rw-r--r-- 1 tulaneID groupID     0 Apr  1 17:54 log_runner_3303151.err
-rw-r--r-- 1 tulaneID groupID   491 Apr  1 17:54 log_submitter_3303148.err
-rw-r--r-- 1 tulaneID groupID   761 Apr  1 17:54 log_submitter_3303148.out
-rw-r--r-- 1 tulaneID groupID    40 Apr  1 17:54 worker_ids.txt
-rw-r--r-- 1 tulaneID groupID    11 Apr  1 17:53 log_runner_3303149.err
-rw-r--r-- 1 tulaneID groupID   956 Apr  1 17:53 log_runner_3303149.out
-rw-r--r-- 1 tulaneID groupID   461 Apr  1 17:51 log_submitter_3303146.err
-rw-r--r-- 1 tulaneID groupID   761 Apr  1 17:51 log_submitter_3303146.out
-rw-r--r-- 1 tulaneID groupID    11 Apr  1 17:51 log_runner_3303147.err
-rw-r--r-- 1 tulaneID groupID   956 Apr  1 17:51 log_runner_3303147.out
-rw-r--r-- 1 tulaneID groupID   491 Apr  1 17:49 log_submitter_3303142.err
-rw-r--r-- 1 tulaneID groupID   761 Apr  1 17:49 log_submitter_3303142.out
-rw-r--r-- 1 tulaneID groupID    11 Apr  1 17:48 log_runner_3303143.err
-rw-r--r-- 1 tulaneID groupID   956 Apr  1 17:48 log_runner_3303143.out
-rw-r--r-- 1 tulaneID groupID   768 Apr  1 17:46 log_submitter_3303140.out
-rw-r--r-- 1 tulaneID groupID   491 Apr  1 17:46 log_submitter_3303140.err
-rw-r--r-- 1 tulaneID groupID    11 Apr  1 17:46 log_runner_3303141.err
-rw-r--r-- 1 tulaneID groupID   948 Apr  1 17:46 log_runner_3303141.out

Listing of the final checkpoint submitter log - completed workflow

The following is the listing of the final log file, log_submitter_3303154.out, generated by checkpoint_submitter.sh ending with "Workflow complete. Exiting submitter."

[tulaneID@cypress2 ~]$cat log_submitter_3303154.out
Info[20260401-17:57:08]: Start on cypress01-066; JOBID=3303154
Info[20260401-17:57:08]: Submitter settings:
Info[20260401-17:57:08]: CKPT_PATH=state_iter_bash.txt
Info[20260401-17:57:08]: MAX_ITER=500
Info[20260401-17:57:08]: RUNNER_SCRIPT=checkpoint_runner.sh
Info[20260401-17:57:08]: SUBMITTER_MARGIN_SEC=60
Info[20260401-17:57:08]: SUBMITTER_TIME_LIMIT=3:30
Info[20260401-17:57:08]: Loaded state: submitter_cycle=5
Info[20260401-17:57:08]: Submitter cycle: 5
Info[20260401-17:57:08]: Current checkpoint iteration: 501
Info[20260401-17:57:08]: Reached MAX_ITER=500; running final analytics.
Info[20260401-17:57:08]: Running analytics aggregate_usage.py ...
Wrote worker_usage.csv
     JobID  ElapsedSeconds   TotalCPU  ... ReqMem       NodeList      State
0  3303141             120  00:00.552  ...  512Mn  cypress01-066     FAILED
1  3303143             120  00:00.563  ...  512Mn  cypress01-066     FAILED
2  3303147             120  00:00.562  ...  512Mn  cypress01-066     FAILED
3  3303149             120  00:00.563  ...  512Mn  cypress01-066     FAILED
4  3303151              22  00:00.347  ...  512Mn  cypress01-066  COMPLETED

[5 rows x 9 columns]
Info[20260401-17:57:09]: Running analytics plot_progress.py ...
Saved progress_with_anomalies.png
Saved resource_trend.png
Saved anomaly_details.csv
Saved anomaly_stats_table.png
Info[20260401-17:57:13]: Workflow complete. Exiting submitter.

Listing of the final checkpoint runner log - completed application

The following is the listing of the final log file, log_runner_3303151.out, generated by checkpoint_runner.sh ending with message "Application finished all iterations."

[tulaneID@cypress2 ~]$cat log_runner_3303151.out
Info[20260401-17:54:37]: Start on cypress01-066; JOBID=3303151
Info[20260401-17:54:37]: Runner settings:
Info[20260401-17:54:37]: APP_CMD=./checkpoint_signal_iter.sh
Info[20260401-17:54:37]: CHECKPOINT_EVERY=20
Info[20260401-17:54:37]: CKPT_PATH=state_iter_bash.txt
Info[20260401-17:54:37]: LAUNCH_MODE=direct
Info[20260401-17:54:37]: MAX_ITER=500
Info[20260401-17:54:37]: MODULE_LIST=anaconda3/2023.07
Info[20260401-17:54:37]: RUNNER_MARGIN_SEC=60
Info[20260401-17:54:37]: RUNNER_TIME_LIMIT=00:03:00
Info[20260401-17:54:37]: SRUN_ARGS=-n 1
Info[20260401-17:54:37]: Loading modules.
Resuming from i=480 (every 20, MAX_ITER=500)
[periodic/iter] saved i=500
Reached i=501 > 500; exiting 0
Info[20260401-17:54:59]: Program exit code (from timeout wrapper): 0
Info[20260401-17:54:59]: Application finished all iterations.

Listings of post-workflow analysis output files

The following is the complete, annotated listing of all the generated .csv and .txt files

[tulaneID@cypress2 ~]$tail -n +1 *.csv *.txt
==> anomaly_details.csv <==
JobID,StartTime,ElapsedSeconds,MaxRSS_MiB,TimeZ,MemModZ,AnomalyType
3303141,2026-04-01 17:44:05,120.0,3.726959228515625,0.49999999999999983,0.0,none
3303143,2026-04-01 17:46:46,120.0,3.73077392578125,0.49999999999999983,0.0,none
3303147,2026-04-01 17:49:26,120.0,3.73077392578125,0.49999999999999983,0.0,none
3303149,2026-04-01 17:51:57,120.0,3.73077392578125,0.49999999999999983,0.0,none
3303151,2026-04-01 17:54:37,22.0,3.7078857421875,-2.0,0.0,none

==> worker_usage.csv <==
JobID,ElapsedSeconds,TotalCPU,MaxRSS,MaxRSS_bytes,MaxRSS_MiB,ReqMem,NodeList,State
3303141,120,00:00.552,3908K,3908000.0,3.726959228515625,512Mn,cypress01-066,FAILED
3303143,120,00:00.563,3912K,3912000.0,3.73077392578125,512Mn,cypress01-066,FAILED
3303147,120,00:00.562,3912K,3912000.0,3.73077392578125,512Mn,cypress01-066,FAILED
3303149,120,00:00.563,3912K,3912000.0,3.73077392578125,512Mn,cypress01-066,FAILED
3303151,22,00:00.347,3888K,3888000.0,3.7078857421875,512Mn,cypress01-066,COMPLETED

==> state_iter_bash.txt <==
501

==> submitter_state.txt <==
submitter_cycle=5

==> worker_ids.txt <==
3303141
3303143
3303147
3303149
3303151

Post-workflow analysis plots

Anomaly stats, compute time and memory usage

Here is the plot, anomaly_stats_table.png, showing the table of anomalies, according to | Z | > 3.5 for execution time and modZ > 3.5 for MaxRSS (SLURM max job memory usage), where Z and modZ are as follows.

Z = (value - mean) / (standard deviation)
modZ = 0.6745 * | x - median | / (Median Absolute Deviation)

See also ​NIST - Detection of Outliers.

Compute progress

Here is the plot, progress_with_anomalies.png, showing Checkpoint Progress and MaxRSS (SLURM max job memory usage) vs Wall Time. Anomalies, if any, would be overplotted in red.

Resource trend

Here is the plot, resource_trend.png, showing Elapsed (time) and MaxRSS vs. Worker job index.