BASH Checkpointing Example

BASH Checkpointing Application

Here is an example of a checkpointing application implemented in BASH script.

By default, the BASH checkpointing application behaves as follows.

• checkpointing every 20 application iterations (one second per iteration) and
• running a total of 500 iterations.

Running the BASH checkpointing example with defaults

To run the application on Cypress, perform the following.

1. Create your own versions of the following files in your current directory on Cypress. (For file editing with nano on Cypress, see ​File Editing Example.)

• checkpoint_submitter.sh
• checkpoint_runner.sh
• aggregate_usage.py
• plot_progress.py

2. Create your own version of the following BASH application script file checkpoint_signal_iter.sh.

#!/usr/bin/env bash
# checkpoint_signal_iter.sh
set -euo pipefail

CKPT="${CKPT_PATH:-state_iter.txt}"
EVERY="${CHECKPOINT_EVERY:-20}"
MAX_ITER="${MAX_ITER:-500}"
i=0

save() {
  # atomic write: tmp + mv
  local val="$1"
  local dir; dir="$(dirname "$(readlink -f "$CKPT")")"
  mkdir -p "$dir"
  local tmp; tmp="$(mktemp -p "$dir" .ckpt.XXXXXX)"
  printf '%s\n' "$val" > "$tmp"
  # Optional (broad) flush for older coreutils: uncomment if you want it
  # sync   # (no -d on coreutils 8.4)
  mv -f "$tmp" "$CKPT"
}

# --- load checkpoint if present ---
if [[ -f "$CKPT" ]]; then
  if [[ "$(cat "$CKPT" 2>/dev/null | tr -d '\r\n' | sed -e 's/[^0-9]//g')" != "" ]]; then
    i="$(cat "$CKPT")"
  fi
fi

# --- handle TERM: save current i and exit(99) to trigger requeue ---
term_handler() {
  echo "Bash SIGTERM: saving i=${i} and exiting 99"
  save "$i"
  exit 99
}
trap 'term_handler' TERM

echo "Resuming from i=${i} (every ${EVERY}, MAX_ITER=${MAX_ITER})"
while true; do
  i=$(( i + 1 ))
  sleep 1

  if (( i % EVERY == 0 )); then
    save "$i"
    echo "[periodic/iter] saved i=${i}"
  fi

  if (( i > MAX_ITER )); then
    echo "Reached i=${i} > ${MAX_ITER}; exiting 0"
    save "$i"
    exit 0
  fi
done

3. Change permissions on the BASH application script, checkpoint_signal_iter.sh, making the script executable via the following chmod command. (See chmod.)

[tulaneID@cypress1 ~]$ chmod u+x checkpoint_signal_iter.sh

4. Submit the job via the following command.

[tulaneID@cypress1 ~]$ APP_CMD="./checkpoint_signal_iter.sh" \
CKPT_PATH=state_iter_bash.txt \
sbatch checkpoint_submitter.sh

5. You can monitor job outputs of both the checkpoint_submitter.sh and the checkpoint_runner.sh jobs as they are updated by executing the following tail command using the -F option to retry once per second, then using Ctrl+C to exit the retry process entirely.

[tulaneID@cypress1 ~]$ tail -F log*

Overriding the default settings

To customize your workflow, see Overriding default settings for overriding the default settings in the checkpoint submitter according to your needs.

Sample run outputs

List of output files

Here is a list of all output files from a sample run sorted by time, latest-to-earliest

[tulaneID@cypress2 ~]$ls -lt *.csv *.err *.out *.png *.txt
-rw-r--r-- 1 tulaneID groupID   452 Apr  1 17:57 anomaly_details.csv
-rw-r--r-- 1 tulaneID groupID 26376 Apr  1 17:57 anomaly_stats_table.png
-rw-r--r-- 1 tulaneID groupID  1402 Apr  1 17:57 log_submitter_3303154.out
-rw-r--r-- 1 tulaneID groupID 40571 Apr  1 17:57 resource_trend.png
-rw-r--r-- 1 tulaneID groupID 52107 Apr  1 17:57 progress_with_anomalies.png
-rw-r--r-- 1 tulaneID groupID   495 Apr  1 17:57 worker_usage.csv
-rw-r--r-- 1 tulaneID groupID   461 Apr  1 17:57 log_submitter_3303150.err
-rw-r--r-- 1 tulaneID groupID   761 Apr  1 17:57 log_submitter_3303150.out
-rw-r--r-- 1 tulaneID groupID     0 Apr  1 17:57 log_submitter_3303154.err
-rw-r--r-- 1 tulaneID groupID    18 Apr  1 17:57 submitter_state.txt
-rw-r--r-- 1 tulaneID groupID   818 Apr  1 17:54 log_runner_3303151.out
-rw------- 1 tulaneID groupID     4 Apr  1 17:54 state_iter_bash.txt
-rw-r--r-- 1 tulaneID groupID     0 Apr  1 17:54 log_runner_3303151.err
-rw-r--r-- 1 tulaneID groupID   491 Apr  1 17:54 log_submitter_3303148.err
-rw-r--r-- 1 tulaneID groupID   761 Apr  1 17:54 log_submitter_3303148.out
-rw-r--r-- 1 tulaneID groupID    40 Apr  1 17:54 worker_ids.txt
-rw-r--r-- 1 tulaneID groupID    11 Apr  1 17:53 log_runner_3303149.err
-rw-r--r-- 1 tulaneID groupID   956 Apr  1 17:53 log_runner_3303149.out
-rw-r--r-- 1 tulaneID groupID   461 Apr  1 17:51 log_submitter_3303146.err
-rw-r--r-- 1 tulaneID groupID   761 Apr  1 17:51 log_submitter_3303146.out
-rw-r--r-- 1 tulaneID groupID    11 Apr  1 17:51 log_runner_3303147.err
-rw-r--r-- 1 tulaneID groupID   956 Apr  1 17:51 log_runner_3303147.out
-rw-r--r-- 1 tulaneID groupID   491 Apr  1 17:49 log_submitter_3303142.err
-rw-r--r-- 1 tulaneID groupID   761 Apr  1 17:49 log_submitter_3303142.out
-rw-r--r-- 1 tulaneID groupID    11 Apr  1 17:48 log_runner_3303143.err
-rw-r--r-- 1 tulaneID groupID   956 Apr  1 17:48 log_runner_3303143.out
-rw-r--r-- 1 tulaneID groupID   768 Apr  1 17:46 log_submitter_3303140.out
-rw-r--r-- 1 tulaneID groupID   491 Apr  1 17:46 log_submitter_3303140.err
-rw-r--r-- 1 tulaneID groupID    11 Apr  1 17:46 log_runner_3303141.err
-rw-r--r-- 1 tulaneID groupID   948 Apr  1 17:46 log_runner_3303141.out

Sample checkpoint submitter logs

Here are the first and last instances from among the multiple log files generated by the sequence of jobs running the job script checkpoint_submitter.sh.

The first job in the sequence was submitted by the user, and all other jobs thereafter in that sequence were submitted by the job script itself via self-restarting.

Listing of the initial checkpoint submitter log - starting workflow

Here is the listing of the first log file, log_submitter_3303140.out, generated by checkpoint_submitter.sh ending with the message "Submitter approaching timeout -> restarting self. Submitted batch job 3303142."

[tulaneID@cypress2 ~]$cat log_submitter_3303140.out
Info[20260401-17:44:05]: Start on cypress01-066; JOBID=3303140
Info[20260401-17:44:05]: Submitter settings:
Info[20260401-17:44:05]: CKPT_PATH=state_iter_bash.txt
Info[20260401-17:44:05]: MAX_ITER=500
Info[20260401-17:44:05]: RUNNER_SCRIPT=checkpoint_runner.sh
Info[20260401-17:44:05]: SUBMITTER_MARGIN_SEC=60
Info[20260401-17:44:05]: SUBMITTER_TIME_LIMIT=3:30
Info[20260401-17:44:05]: No previous state found; starting fresh.
Info[20260401-17:44:05]: Submitter cycle: 0
Info[20260401-17:44:05]: Current checkpoint iteration: 0
Info[20260401-17:44:05]: Submitted worker: 3303141
Info[20260401-17:44:05]: Waiting on first of worker 3303141 to finish or self-restart.
Info[20260401-17:46:46]: Submitter approaching timeout -> restarting self
Submitted batch job 3303142

Listing of the final checkpoint submitter log - completed workflow

Here is the listing of the final log file, log_submitter_3303154.out, generated by checkpoint_submitter.sh ending with the message "Workflow complete. Exiting submitter."

[tulaneID@cypress2 ~]$cat log_submitter_3303154.out
Info[20260401-17:57:08]: Start on cypress01-066; JOBID=3303154
Info[20260401-17:57:08]: Submitter settings:
Info[20260401-17:57:08]: CKPT_PATH=state_iter_bash.txt
Info[20260401-17:57:08]: MAX_ITER=500
Info[20260401-17:57:08]: RUNNER_SCRIPT=checkpoint_runner.sh
Info[20260401-17:57:08]: SUBMITTER_MARGIN_SEC=60
Info[20260401-17:57:08]: SUBMITTER_TIME_LIMIT=3:30
Info[20260401-17:57:08]: Loaded state: submitter_cycle=5
Info[20260401-17:57:08]: Submitter cycle: 5
Info[20260401-17:57:08]: Current checkpoint iteration: 501
Info[20260401-17:57:08]: Reached MAX_ITER=500; running final analytics.
Info[20260401-17:57:08]: Running analytics aggregate_usage.py ...
Wrote worker_usage.csv
     JobID  ElapsedSeconds   TotalCPU  ... ReqMem       NodeList      State
0  3303141             120  00:00.552  ...  512Mn  cypress01-066     FAILED
1  3303143             120  00:00.563  ...  512Mn  cypress01-066     FAILED
2  3303147             120  00:00.562  ...  512Mn  cypress01-066     FAILED
3  3303149             120  00:00.563  ...  512Mn  cypress01-066     FAILED
4  3303151              22  00:00.347  ...  512Mn  cypress01-066  COMPLETED

[5 rows x 9 columns]
Info[20260401-17:57:09]: Running analytics plot_progress.py ...
Saved progress_with_anomalies.png
Saved resource_trend.png
Saved anomaly_details.csv
Saved anomaly_stats_table.png
Info[20260401-17:57:13]: Workflow complete. Exiting submitter.

Sample checkpoint runner logs

Here are the first and last output logs of the sequence of jobs generated by the job script checkpoint_runner.sh.

All of the jobs in that sequence were submitted by the checkpoint_submitter.sh job script.

Listing of the initial checkpoint runner log - starting application

Here is the listing of the intial log file, log_runner_3303141.out, generated by the job script checkpoint_runner.sh ending with message "Timeout + checkpoint advance detected (0->120)."

[tulaneID@cypress2 ~]$cat log_runner_3303141.out 
Info[20260401-17:44:05]: Start on cypress01-066; JOBID=3303141
Info[20260401-17:44:05]: Runner settings:
Info[20260401-17:44:05]: APP_CMD=./checkpoint_signal_iter.sh
Info[20260401-17:44:05]: CHECKPOINT_EVERY=20
Info[20260401-17:44:05]: CKPT_PATH=state_iter_bash.txt
Info[20260401-17:44:05]: LAUNCH_MODE=direct
Info[20260401-17:44:05]: MAX_ITER=500
Info[20260401-17:44:05]: MODULE_LIST=anaconda3/2023.07
Info[20260401-17:44:05]: RUNNER_MARGIN_SEC=60
Info[20260401-17:44:05]: RUNNER_TIME_LIMIT=00:03:00
Info[20260401-17:44:05]: SRUN_ARGS=-n 1
Info[20260401-17:44:05]: Loading modules.
Resuming from i=0 (every 20, MAX_ITER=500)
[periodic/iter] saved i=20
[periodic/iter] saved i=40
[periodic/iter] saved i=60
[periodic/iter] saved i=80
[periodic/iter] saved i=100
Bash SIGTERM: saving i=120 and exiting 99
Info[20260401-17:46:05]: Program exit code (from timeout wrapper): 124
Info[20260401-17:46:05]: Timeout + checkpoint advance detected (0->120).

Listing of the final checkpoint runner log - completed application

Here is the listing of the final log file, log_runner_3303151.out, generated by checkpoint_runner.sh ending with message "Application finished all iterations."

[tulaneID@cypress2 ~]$cat log_runner_3303151.out
Info[20260401-17:54:37]: Start on cypress01-066; JOBID=3303151
Info[20260401-17:54:37]: Runner settings:
Info[20260401-17:54:37]: APP_CMD=./checkpoint_signal_iter.sh
Info[20260401-17:54:37]: CHECKPOINT_EVERY=20
Info[20260401-17:54:37]: CKPT_PATH=state_iter_bash.txt
Info[20260401-17:54:37]: LAUNCH_MODE=direct
Info[20260401-17:54:37]: MAX_ITER=500
Info[20260401-17:54:37]: MODULE_LIST=anaconda3/2023.07
Info[20260401-17:54:37]: RUNNER_MARGIN_SEC=60
Info[20260401-17:54:37]: RUNNER_TIME_LIMIT=00:03:00
Info[20260401-17:54:37]: SRUN_ARGS=-n 1
Info[20260401-17:54:37]: Loading modules.
Resuming from i=480 (every 20, MAX_ITER=500)
[periodic/iter] saved i=500
Reached i=501 > 500; exiting 0
Info[20260401-17:54:59]: Program exit code (from timeout wrapper): 0
Info[20260401-17:54:59]: Application finished all iterations.

Listings of post-workflow analysis output files

The following is the complete, annotated listing of all the generated .csv and .txt files

[tulaneID@cypress2 ~]$tail -n +1 *.csv *.txt
==> anomaly_details.csv <==
JobID,StartTime,ElapsedSeconds,MaxRSS_MiB,TimeZ,MemModZ,AnomalyType
3303141,2026-04-01 17:44:05,120.0,3.726959228515625,0.49999999999999983,0.0,none
3303143,2026-04-01 17:46:46,120.0,3.73077392578125,0.49999999999999983,0.0,none
3303147,2026-04-01 17:49:26,120.0,3.73077392578125,0.49999999999999983,0.0,none
3303149,2026-04-01 17:51:57,120.0,3.73077392578125,0.49999999999999983,0.0,none
3303151,2026-04-01 17:54:37,22.0,3.7078857421875,-2.0,0.0,none

==> worker_usage.csv <==
JobID,ElapsedSeconds,TotalCPU,MaxRSS,MaxRSS_bytes,MaxRSS_MiB,ReqMem,NodeList,State
3303141,120,00:00.552,3908K,3908000.0,3.726959228515625,512Mn,cypress01-066,FAILED
3303143,120,00:00.563,3912K,3912000.0,3.73077392578125,512Mn,cypress01-066,FAILED
3303147,120,00:00.562,3912K,3912000.0,3.73077392578125,512Mn,cypress01-066,FAILED
3303149,120,00:00.563,3912K,3912000.0,3.73077392578125,512Mn,cypress01-066,FAILED
3303151,22,00:00.347,3888K,3888000.0,3.7078857421875,512Mn,cypress01-066,COMPLETED

==> state_iter_bash.txt <==
501

==> submitter_state.txt <==
submitter_cycle=5

==> worker_ids.txt <==
3303141
3303143
3303147
3303149
3303151

Post-workflow analysis plots

Background

For background on the following plots, see Plotting execution time and memory usage.

Table of anomalies, compute time and memory usage

Here is the plot, anomaly_stats_table.png, showing the table of anomalies, if any, from among all of the checkpoint_runner.sh jobs.

Compute progress

Here is the plot, progress_with_anomalies.png, showing Checkpoint Progress and MaxRSS (SLURM max job memory usage) vs Wall Time.

Execution time and memory usage trends

Here is the plot, resource_trend.png, showing Elapsed (time) and MaxRSS vs. Worker job index.