BASH Checkpointing Example

BASH Checkpointing Application

Here is an example of a checkpointing application implemented in BASH script.

To run the example, see Running the BASH checkpointing example on Cypress below.

checkpoint_signal_iter.sh

#!/usr/bin/env bash
# checkpoint_signal_iter.sh
set -euo pipefail

CKPT="${CKPT_PATH:-state_iter.txt}"
EVERY="${CHECKPOINT_EVERY:-20}"
MAX_ITER="${MAX_ITER:-500}"
i=0

save() {
  # atomic write: tmp + mv
  local val="$1"
  local dir; dir="$(dirname "$(readlink -f "$CKPT")")"
  mkdir -p "$dir"
  local tmp; tmp="$(mktemp -p "$dir" .ckpt.XXXXXX)"
  printf '%s\n' "$val" > "$tmp"
  # Optional (broad) flush for older coreutils: uncomment if you want it
  # sync   # (no -d on coreutils 8.4)
  mv -f "$tmp" "$CKPT"
}

# --- load checkpoint if present ---
if [[ -f "$CKPT" ]]; then
  if [[ "$(cat "$CKPT" 2>/dev/null | tr -d '\r\n' | sed -e 's/[^0-9]//g')" != "" ]]; then
    i="$(cat "$CKPT")"
  fi
fi

# --- handle TERM: save current i and exit(99) to trigger requeue ---
term_handler() {
  echo "Bash SIGTERM: saving i=${i} and exiting 99"
  save "$i"
  exit 99
}
trap 'term_handler' TERM

echo "Resuming from i=${i} (every ${EVERY}, MAX_ITER=${MAX_ITER})"
while true; do
  i=$(( i + 1 ))
  sleep 1

  if (( i % EVERY == 0 )); then
    save "$i"
    echo "[periodic/iter] saved i=${i}"
  fi

  if (( i > MAX_ITER )); then
    echo "Reached i=${i} > ${MAX_ITER}; exiting 0"
    save "$i"
    exit 0
  fi
done

Running the BASH checkpointing example on Cypress

To run the BASH checkpointing job example, defaulting to checkpointing every 20 application iterations and a total of 500 iterations, perform the following.

1. Create your own versions of the following files in your current directory on Cypress. (For file editing with nano on Cypress, see ​File Editing Example.)

• checkpoint_submitter.sh
• checkpoint_runner.sh
• aggregate_usage.py
• plot_progress.py
• checkpoint_signal_iter.sh

2. Change permissions on the BASH application script, checkpoint_signal_iter.sh executable via the following command.

[tulaneID@cypress1 ~]$ chmod u+x checkpoint_signal_iter.sh

3. Submit the job via the following command.

[tulaneID@cypress1 ~]$ APP_CMD="./checkpoint_signal_iter.sh" \
CKPT_PATH=state_iter_bash.txt \
sbatch checkpoint_runner.sh

4. You can monitor the jobs' outputs as they are updated by repeatedly executing the following command, using Ctrl+C to exit files as they finish updating.

[tulaneID@cypress1 ~]$ tail -f log_*

Overriding the default settings

To customize your workflow, see Overriding default settings for overriding the default settings in the checkpoint submitter according to your needs.

Sample run outputs

List of output files

Here is a list of all output files from a sample run sorted by time, latest-to-earliest

[tulaneID@cypress2 ~]$ls -lt *.csv *.err *.out *.png *.txt
-rw-r--r-- 1 tulaneID groupID   452 Apr  1 17:57 anomaly_details.csv
-rw-r--r-- 1 tulaneID groupID 26376 Apr  1 17:57 anomaly_stats_table.png
-rw-r--r-- 1 tulaneID groupID  1402 Apr  1 17:57 log_submitter_3303154.out
-rw-r--r-- 1 tulaneID groupID 40571 Apr  1 17:57 resource_trend.png
-rw-r--r-- 1 tulaneID groupID 52107 Apr  1 17:57 progress_with_anomalies.png
-rw-r--r-- 1 tulaneID groupID   495 Apr  1 17:57 worker_usage.csv
-rw-r--r-- 1 tulaneID groupID   461 Apr  1 17:57 log_submitter_3303150.err
-rw-r--r-- 1 tulaneID groupID   761 Apr  1 17:57 log_submitter_3303150.out
-rw-r--r-- 1 tulaneID groupID     0 Apr  1 17:57 log_submitter_3303154.err
-rw-r--r-- 1 tulaneID groupID    18 Apr  1 17:57 submitter_state.txt
-rw-r--r-- 1 tulaneID groupID   818 Apr  1 17:54 log_runner_3303151.out
-rw------- 1 tulaneID groupID     4 Apr  1 17:54 state_iter_bash.txt
-rw-r--r-- 1 tulaneID groupID     0 Apr  1 17:54 log_runner_3303151.err
-rw-r--r-- 1 tulaneID groupID   491 Apr  1 17:54 log_submitter_3303148.err
-rw-r--r-- 1 tulaneID groupID   761 Apr  1 17:54 log_submitter_3303148.out
-rw-r--r-- 1 tulaneID groupID    40 Apr  1 17:54 worker_ids.txt
-rw-r--r-- 1 tulaneID groupID    11 Apr  1 17:53 log_runner_3303149.err
-rw-r--r-- 1 tulaneID groupID   956 Apr  1 17:53 log_runner_3303149.out
-rw-r--r-- 1 tulaneID groupID   461 Apr  1 17:51 log_submitter_3303146.err
-rw-r--r-- 1 tulaneID groupID   761 Apr  1 17:51 log_submitter_3303146.out
-rw-r--r-- 1 tulaneID groupID    11 Apr  1 17:51 log_runner_3303147.err
-rw-r--r-- 1 tulaneID groupID   956 Apr  1 17:51 log_runner_3303147.out
-rw-r--r-- 1 tulaneID groupID   491 Apr  1 17:49 log_submitter_3303142.err
-rw-r--r-- 1 tulaneID groupID   761 Apr  1 17:49 log_submitter_3303142.out
-rw-r--r-- 1 tulaneID groupID    11 Apr  1 17:48 log_runner_3303143.err
-rw-r--r-- 1 tulaneID groupID   956 Apr  1 17:48 log_runner_3303143.out
-rw-r--r-- 1 tulaneID groupID   768 Apr  1 17:46 log_submitter_3303140.out
-rw-r--r-- 1 tulaneID groupID   491 Apr  1 17:46 log_submitter_3303140.err
-rw-r--r-- 1 tulaneID groupID    11 Apr  1 17:46 log_runner_3303141.err
-rw-r--r-- 1 tulaneID groupID   948 Apr  1 17:46 log_runner_3303141.out

Listing of the final checkpoint submitter log - completed workflow

The following is the listing of the final log file, log_submitter_3303154.out, generated by checkpoint_submitter.sh ending with "Workflow complete. Exiting submitter."

[tulaneID@cypress2 ~]$cat log_submitter_3303154.out
Info[20260401-17:57:08]: Start on cypress01-066; JOBID=3303154
Info[20260401-17:57:08]: Submitter settings:
Info[20260401-17:57:08]: CKPT_PATH=state_iter_bash.txt
Info[20260401-17:57:08]: MAX_ITER=500
Info[20260401-17:57:08]: RUNNER_SCRIPT=checkpoint_runner.sh
Info[20260401-17:57:08]: SUBMITTER_MARGIN_SEC=60
Info[20260401-17:57:08]: SUBMITTER_TIME_LIMIT=3:30
Info[20260401-17:57:08]: Loaded state: submitter_cycle=5
Info[20260401-17:57:08]: Submitter cycle: 5
Info[20260401-17:57:08]: Current checkpoint iteration: 501
Info[20260401-17:57:08]: Reached MAX_ITER=500; running final analytics.
Info[20260401-17:57:08]: Running analytics aggregate_usage.py ...
Wrote worker_usage.csv
     JobID  ElapsedSeconds   TotalCPU  ... ReqMem       NodeList      State
0  3303141             120  00:00.552  ...  512Mn  cypress01-066     FAILED
1  3303143             120  00:00.563  ...  512Mn  cypress01-066     FAILED
2  3303147             120  00:00.562  ...  512Mn  cypress01-066     FAILED
3  3303149             120  00:00.563  ...  512Mn  cypress01-066     FAILED
4  3303151              22  00:00.347  ...  512Mn  cypress01-066  COMPLETED

[5 rows x 9 columns]
Info[20260401-17:57:09]: Running analytics plot_progress.py ...
Saved progress_with_anomalies.png
Saved resource_trend.png
Saved anomaly_details.csv
Saved anomaly_stats_table.png
Info[20260401-17:57:13]: Workflow complete. Exiting submitter.

Listing of the final checkpoint runner log - completed application

The following is the listing of the final log file, log_runner_3303151.out, generated by checkpoint_runner.sh ending with message "Application finished all iterations."

[tulaneID@cypress2 ~]$cat log_runner_3303151.out
Info[20260401-17:54:37]: Start on cypress01-066; JOBID=3303151
Info[20260401-17:54:37]: Runner settings:
Info[20260401-17:54:37]: APP_CMD=./checkpoint_signal_iter.sh
Info[20260401-17:54:37]: CHECKPOINT_EVERY=20
Info[20260401-17:54:37]: CKPT_PATH=state_iter_bash.txt
Info[20260401-17:54:37]: LAUNCH_MODE=direct
Info[20260401-17:54:37]: MAX_ITER=500
Info[20260401-17:54:37]: MODULE_LIST=anaconda3/2023.07
Info[20260401-17:54:37]: RUNNER_MARGIN_SEC=60
Info[20260401-17:54:37]: RUNNER_TIME_LIMIT=00:03:00
Info[20260401-17:54:37]: SRUN_ARGS=-n 1
Info[20260401-17:54:37]: Loading modules.
Resuming from i=480 (every 20, MAX_ITER=500)
[periodic/iter] saved i=500
Reached i=501 > 500; exiting 0
Info[20260401-17:54:59]: Program exit code (from timeout wrapper): 0
Info[20260401-17:54:59]: Application finished all iterations.

Listings of post-workflow analysis output files

The following is the complete, annotated listing of all the generated .csv and .txt files

[tulaneID@cypress2 ~]$tail -n +1 *.csv *.txt
==> anomaly_details.csv <==
JobID,StartTime,ElapsedSeconds,MaxRSS_MiB,TimeZ,MemModZ,AnomalyType
3303141,2026-04-01 17:44:05,120.0,3.726959228515625,0.49999999999999983,0.0,none
3303143,2026-04-01 17:46:46,120.0,3.73077392578125,0.49999999999999983,0.0,none
3303147,2026-04-01 17:49:26,120.0,3.73077392578125,0.49999999999999983,0.0,none
3303149,2026-04-01 17:51:57,120.0,3.73077392578125,0.49999999999999983,0.0,none
3303151,2026-04-01 17:54:37,22.0,3.7078857421875,-2.0,0.0,none

==> worker_usage.csv <==
JobID,ElapsedSeconds,TotalCPU,MaxRSS,MaxRSS_bytes,MaxRSS_MiB,ReqMem,NodeList,State
3303141,120,00:00.552,3908K,3908000.0,3.726959228515625,512Mn,cypress01-066,FAILED
3303143,120,00:00.563,3912K,3912000.0,3.73077392578125,512Mn,cypress01-066,FAILED
3303147,120,00:00.562,3912K,3912000.0,3.73077392578125,512Mn,cypress01-066,FAILED
3303149,120,00:00.563,3912K,3912000.0,3.73077392578125,512Mn,cypress01-066,FAILED
3303151,22,00:00.347,3888K,3888000.0,3.7078857421875,512Mn,cypress01-066,COMPLETED

==> state_iter_bash.txt <==
501

==> submitter_state.txt <==
submitter_cycle=5

==> worker_ids.txt <==
3303141
3303143
3303147
3303149
3303151

Post-workflow analysis plots

Anomaly stats, compute time and memory usage

Here is the plot, anomaly_stats_table.png, showing the table of anomalies, according to | Z | > 3.5 for execution time and modZ > 3.5 for MaxRSS (SLURM max job memory usage), where Z and modZ are as follows.

Z = (value - mean) / (standard deviation)
modZ = 0.6745 * | x - median | / (Median Absolute Deviation)

See also ​NIST - Detection of Outliers.

Compute progress

Here is the plot, progress_with_anomalies.png, showing Checkpoint Progress and MaxRSS (SLURM max job memory usage) vs Wall Time. Anomalies, if any, would be overplotted in red.

Resource trend

Here is the plot, resource_trend.png, showing Elapsed (time) and MaxRSS vs. Worker job index.