R Checkpointing Example

Here is an example of a checkpointing application implemented in R script.

By default, the R checkpointing application behaves as follows.

• checkpointing every 20 application iterations (one second per iteration) and
• running a total of 500 iterations.

Running the R checkpointing example with defaults

To run the application on Cypress, perform the following.

1. Create your own versions of the following files in your current directory on Cypress. (For file editing with nano on Cypress, see ​File Editing Example.)

• checkpoint_submitter.sh
• checkpoint_runner.sh
• aggregate_usage.py
• plot_progress.py

2. Create your own version of the following R application script file checkpoint_signal_R.sh.

#!/usr/bin/env Rscript
# checkpoint_signal_iter.R  (requires 'sigterm' package)
suppressWarnings(suppressMessages({
  if (!requireNamespace("sigterm", quietly = TRUE)) {
    message("ERROR: 'sigterm' package not installed; install via remotes/devtools.")
    quit(save="no", status=2L)
  }
}))

get_env <- function(k, d) { v <- Sys.getenv(k, unset=NA); if (is.na(v) || v=="") d else v }
CKPT <- get_env("CKPT_PATH", "state_iter.txt")
EVERY <- as.integer(get_env("CHECKPOINT_EVERY", "20"))
MAX_ITER <- as.integer(get_env("MAX_ITER", "500"))

atomic_save <- function(path, val) {
  dir <- dirname(normalizePath(path, mustWork = FALSE))
  if (!dir.exists(dir)) dir.create(dir, recursive = TRUE, showWarnings = FALSE)
  tmp <- tempfile(pattern = ".ckpt.", tmpdir = dir)
  con <- file(tmp, open="wt"); writeLines(as.character(val), con); flush(con); close(con)
  file.rename(tmp, path)
}
load_ckpt <- function(path) {
  if (!file.exists(path)) return(0L)
  txt <- tryCatch(readLines(path, warn = FALSE), error = function(e) "0")
  as.integer(gsub("[^0-9]", "", paste(txt, collapse = "")))
}

library(sigterm)  # installs a SIGTERM handler; poll has_sigterm_flag()

i <- load_ckpt(CKPT)
cat(sprintf("R: Resuming from i=%d (every %d, MAX_ITER=%d)\n", i, EVERY, MAX_ITER)); flush(stdout())

repeat {
  i <- i + 1L
  Sys.sleep(1)

  if (i %% EVERY == 0L) {
    atomic_save(CKPT, i)
    cat(sprintf("[periodic/iter] saved i=%d\n", i)); flush(stdout())
  }

  if (sigterm::has_sigterm_flag()) {
    cat(sprintf("R: SIGTERM detected — saving i=%d and exiting 99\n", i)); flush(stdout())
    atomic_save(CKPT, i)
    quit(save="no", status=99L)
  }

  if (i > MAX_ITER) {
    cat(sprintf("Reached i=%d > %d; exiting 0\n", i, MAX_ITER)); flush(stdout())
    atomic_save(CKPT, i)
    quit(save="no", status=0L)
  }
}

3. Use the steps in the section Alternative for tidyverse for constructing a container image file tidyverse_latest.sif (or tidyverse_4.5.2.sif as in the following) for use with Singularity.

4. Install the required R package sigterm via the following, substituting your own writable R library directory for <R_lib_path>. See Installing R Packages on Cypress for more information on using a writable R library directory.

[tulaneID@cypress1 ~]$idev --partition=centos7
[tulaneID@cypress01-XXX ~]$module load singularity/3.9.0
[tulaneID@cypress01-XXX ~]$singularity shell tidyverse_4.5.2.sif
Singularity> Rscript --version # confirm Rscript is available
Rscript (R) version 4.5.2 (2025-10-31)
Singularity>Rscript -e "devtools::install_github('atheriel/sigterm', lib='<R_lib_path>')"
Singularity>exit # exit the container
[tulaneID@cypress01-XXX ~]$exit # exit the interactive session

5. Submit the job via the following command.

[tulaneID@cypress1 ~]$APP_CMD="singularity exec tidyverse_4.5.2.sif Rscript checkpoint_signal_iter.R" \
MODULE_LIST="singularity/3.9.0" \
CKPT_PATH=state_iter_r.txt \
sbatch checkpoint_submitter.sh

6. You can monitor job outputs of both the checkpoint_submitter.sh and the checkpoint_runner.sh jobs as they are updated by executing the following tail command using the -F option to retry once per second, then using Ctrl+C to exit the retry process entirely.

[tulaneID@cypress1 ~]$ tail -F log*

Overriding the default settings

To customize your workflow, see Overriding default settings for overriding the default settings in the checkpoint submitter according to your needs.

Sample run outputs

List of output files

Here is a list of all output files from a sample run sorted by time, latest-to-earliest

[tulaneID@cypress2 ~]$ls -lt *.csv *.err *.out *.png *.txt
-rw-r--r-- 1 tulaneID groupID 52539 Apr  1 17:07 anomaly_stats_table.png
-rw-r--r-- 1 tulaneID groupID  1399 Apr  1 17:07 log_submitter_3303072.out
-rw-r--r-- 1 tulaneID groupID   468 Apr  1 17:07 anomaly_details.csv
-rw-r--r-- 1 tulaneID groupID   635 Apr  1 17:07 log_submitter_3303072.err
-rw-r--r-- 1 tulaneID groupID 52642 Apr  1 17:07 progress_with_anomalies.png
-rw-r--r-- 1 tulaneID groupID 41081 Apr  1 17:07 resource_trend.png
-rw-r--r-- 1 tulaneID groupID   510 Apr  1 17:07 worker_usage.csv
-rw-r--r-- 1 tulaneID groupID   722 Apr  1 17:07 log_submitter_3303070.out
-rw-r--r-- 1 tulaneID groupID    18 Apr  1 17:07 submitter_state.txt
-rw-r--r-- 1 tulaneID groupID   888 Apr  1 17:04 log_runner_3303071.out
-rw-r--r-- 1 tulaneID groupID     4 Apr  1 17:04 state_iter_r.txt
-rw-r--r-- 1 tulaneID groupID     0 Apr  1 17:04 log_runner_3303071.err
-rw-r--r-- 1 tulaneID groupID   722 Apr  1 17:04 log_submitter_3303067.out
-rw-r--r-- 1 tulaneID groupID     0 Apr  1 17:04 log_submitter_3303070.err
-rw-r--r-- 1 tulaneID groupID    40 Apr  1 17:04 worker_ids.txt
-rw-r--r-- 1 tulaneID groupID  1036 Apr  1 17:03 log_runner_3303068.out
-rw-r--r-- 1 tulaneID groupID     0 Apr  1 17:01 log_runner_3303068.err
-rw-r--r-- 1 tulaneID groupID   722 Apr  1 17:01 log_submitter_3303064.out
-rw-r--r-- 1 tulaneID groupID     0 Apr  1 17:01 log_submitter_3303067.err
-rw-r--r-- 1 tulaneID groupID  1036 Apr  1 17:01 log_runner_3303065.out
-rw-r--r-- 1 tulaneID groupID     0 Apr  1 16:59 log_runner_3303065.err
-rw-r--r-- 1 tulaneID groupID   722 Apr  1 16:59 log_submitter_3303062.out
-rw-r--r-- 1 tulaneID groupID     0 Apr  1 16:59 log_submitter_3303064.err
-rw-r--r-- 1 tulaneID groupID  1036 Apr  1 16:58 log_runner_3303063.out
-rw-r--r-- 1 tulaneID groupID     0 Apr  1 16:56 log_runner_3303063.err
-rw-r--r-- 1 tulaneID groupID   729 Apr  1 16:56 log_submitter_3303060.out
-rw-r--r-- 1 tulaneID groupID     0 Apr  1 16:56 log_submitter_3303062.err
-rw-r--r-- 1 tulaneID groupID  1000 Apr  1 16:56 log_runner_3303061.out
-rw-r--r-- 1 tulaneID groupID     0 Apr  1 16:54 log_runner_3303061.err
-rw-r--r-- 1 tulaneID groupID     0 Apr  1 16:54 log_submitter_3303060.err

Sample checkpoint submitter logs

Here are the first and last instances from among the multiple log files generated by the sequence of jobs running the job script checkpoint_submitter.sh.

The first job in the sequence was submitted by the user, and all other jobs thereafter in that sequence were submitted by the job script itself via self-restarting.

Listing of the initial checkpoint submitter log - starting workflow

Here is the listing of the first log file, log_submitter_3303060.out, generated by checkpoint_submitter.sh ending with the message "Submitter approaching timeout -> restarting self. Submitted batch job 3303142."

[tulaneID@cypress2 ~]$cat log_submitter_3303060.out 
Info[20260401-16:54:06]: Start on cypress01-066; JOBID=3303060
Info[20260401-16:54:06]: Submitter settings:
Info[20260401-16:54:06]: CKPT_PATH=state_iter_r.txt
Info[20260401-16:54:06]: MAX_ITER=500
Info[20260401-16:54:06]: RUNNER_SCRIPT=checkpoint_runner.sh
Info[20260401-16:54:06]: SUBMITTER_MARGIN_SEC=60
Info[20260401-16:54:06]: SUBMITTER_TIME_LIMIT=3:30
Info[20260401-16:54:06]: No previous state found; starting fresh.
Info[20260401-16:54:06]: Submitter cycle: 0
Info[20260401-16:54:06]: Current checkpoint iteration: 0
Info[20260401-16:54:06]: Submitted worker: 3303061
Info[20260401-16:54:06]: Waiting on worker 3303061
Info[20260401-16:56:46]: Submitter approaching timeout -> restarting self
Submitted batch job 3303062

Listing of the final checkpoint submitter log - completed workflow

Here is the listing of the final log file, log_submitter_3303072.out, generated by checkpoint_submitter.sh ending with the message "Workflow complete. Exiting submitter."

[tulaneID@cypress2 ~]$cat log_submitter_3303072.out 
Info[20260401-17:07:08]: Start on cypress01-066; JOBID=3303072
Info[20260401-17:07:08]: Submitter settings:
Info[20260401-17:07:08]: CKPT_PATH=state_iter_r.txt
Info[20260401-17:07:08]: MAX_ITER=500
Info[20260401-17:07:08]: RUNNER_SCRIPT=checkpoint_runner.sh
Info[20260401-17:07:08]: SUBMITTER_MARGIN_SEC=60
Info[20260401-17:07:08]: SUBMITTER_TIME_LIMIT=3:30
Info[20260401-17:07:08]: Loaded state: submitter_cycle=5
Info[20260401-17:07:08]: Submitter cycle: 5
Info[20260401-17:07:08]: Current checkpoint iteration: 501
Info[20260401-17:07:08]: Reached MAX_ITER=500; running final analytics.
Info[20260401-17:07:08]: Running analytics aggregate_usage.py ...
Wrote worker_usage.csv
     JobID  ElapsedSeconds   TotalCPU  ... ReqMem       NodeList      State
0  3303061             121  00:03.487  ...  512Mn  cypress01-066     FAILED
1  3303063             121  00:03.506  ...  512Mn  cypress01-066     FAILED
2  3303065             121  00:03.502  ...  512Mn  cypress01-066     FAILED
3  3303068             121  00:03.484  ...  512Mn  cypress01-066     FAILED
4  3303071              27  00:03.478  ...  512Mn  cypress01-066  COMPLETED

[5 rows x 9 columns]
Info[20260401-17:07:10]: Running analytics plot_progress.py ...
Saved progress_with_anomalies.png
Saved resource_trend.png
Saved anomaly_details.csv
Saved anomaly_stats_table.png
Info[20260401-17:07:14]: Workflow complete. Exiting submitter.

Sample checkpoint runner logs

Here are the first and last output logs of the sequence of jobs generated by the job script checkpoint_runner.sh.

All of the jobs in that sequence were submitted by the checkpoint_submitter.sh job script.

Listing of the initial checkpoint runner log - starting application

Here is the listing of the intial log file, log_runner_3303061.out, generated by the job script checkpoint_runner.sh ending with message "Timeout + checkpoint advance detected (0->120)."

[tulaneID@cypress2 ~]$cat log_runner_3303061.out 
Info[20260401-16:54:06]: Start on cypress01-066; JOBID=3303061
Info[20260401-16:54:06]: Runner settings:
Info[20260401-16:54:06]: APP_CMD=singularity exec tidyverse_4.5.2.sif Rscript checkpoint_signal_iter.R
Info[20260401-16:54:06]: CHECKPOINT_EVERY=20
Info[20260401-16:54:06]: CKPT_PATH=state_iter_r.txt
Info[20260401-16:54:06]: LAUNCH_MODE=direct
Info[20260401-16:54:06]: MAX_ITER=500
Info[20260401-16:54:06]: MODULE_LIST=singularity/3.9.0
Info[20260401-16:54:06]: RUNNER_MARGIN_SEC=60
Info[20260401-16:54:06]: RUNNER_TIME_LIMIT=00:03:00
Info[20260401-16:54:06]: SRUN_ARGS=-n 1
Info[20260401-16:54:06]: Loading modules.
R: Resuming from i=0 (every 20, MAX_ITER=500)
[periodic/iter] saved i=20
[periodic/iter] saved i=40
[periodic/iter] saved i=60
[periodic/iter] saved i=80
[periodic/iter] saved i=100
R: SIGTERM detected — saving i=119 and exiting 99
Info[20260401-16:56:07]: Program exit code (from timeout wrapper): 124
Info[20260401-16:56:07]: Timeout + checkpoint advance detected (0->119).

Listing of the final checkpoint runner log - completed application

Here is the listing of the final log file, log_runner_3303171.out, generated by checkpoint_runner.sh ending with message "Application finished all iterations."

[tulaneID@cypress2 ~]$cat log_runner_3303071.out 
Info[20260401-17:04:28]: Start on cypress01-066; JOBID=3303071
Info[20260401-17:04:28]: Runner settings:
Info[20260401-17:04:28]: APP_CMD=singularity exec tidyverse_4.5.2.sif Rscript checkpoint_signal_iter.R
Info[20260401-17:04:28]: CHECKPOINT_EVERY=20
Info[20260401-17:04:28]: CKPT_PATH=state_iter_r.txt
Info[20260401-17:04:28]: LAUNCH_MODE=direct
Info[20260401-17:04:28]: MAX_ITER=500
Info[20260401-17:04:28]: MODULE_LIST=singularity/3.9.0
Info[20260401-17:04:28]: RUNNER_MARGIN_SEC=60
Info[20260401-17:04:28]: RUNNER_TIME_LIMIT=00:03:00
Info[20260401-17:04:28]: SRUN_ARGS=-n 1
Info[20260401-17:04:28]: Loading modules.
R: Resuming from i=476 (every 20, MAX_ITER=500)
[periodic/iter] saved i=480
[periodic/iter] saved i=500
Reached i=501 > 500; exiting 0
Info[20260401-17:04:55]: Program exit code (from timeout wrapper): 0
Info[20260401-17:04:55]: Application finished all iterations.

Listings of post-workflow analysis output files

The following is the complete, annotated listing of all the generated .csv and .txt files

[tulaneID@cypress2 ~]$tail -n +1 *.csv *.txt
==> anomaly_details.csv <==
JobID,StartTime,ElapsedSeconds,MaxRSS_MiB,TimeZ,MemModZ,AnomalyType
3303061,2026-04-01 16:54:06,121.0,91.64047241210938,0.4999999999999999,0.0,none
3303063,2026-04-01 16:56:46,121.0,93.33038330078124,0.4999999999999999,0.0,none
3303065,2026-04-01 16:59:17,121.0,91.4154052734375,0.4999999999999999,0.0,none
3303068,2026-04-01 17:01:57,121.0,95.40939331054688,0.4999999999999999,0.0,none
3303071,2026-04-01 17:04:28,27.0,44.5709228515625,-2.0,10.857998043052838,memory

==> worker_usage.csv <==
JobID,ElapsedSeconds,TotalCPU,MaxRSS,MaxRSS_bytes,MaxRSS_MiB,ReqMem,NodeList,State
3303061,121,00:03.487,96092K,96092000.0,91.64047241210938,512Mn,cypress01-066,FAILED
3303063,121,00:03.506,97864K,97864000.0,93.33038330078125,512Mn,cypress01-066,FAILED
3303065,121,00:03.502,95856K,95856000.0,91.4154052734375,512Mn,cypress01-066,FAILED
3303068,121,00:03.484,100044K,100044000.0,95.40939331054688,512Mn,cypress01-066,FAILED
3303071,27,00:03.478,46736K,46736000.0,44.5709228515625,512Mn,cypress01-066,COMPLETED

==> state_iter_r.txt <==
501

==> submitter_state.txt <==
submitter_cycle=5

==> worker_ids.txt <==
3303061
3303063
3303065
3303068
3303071

Post-workflow analysis plots

Background

For background on the following plots, see Plotting execution time and memory usage.

Table of anomalies, compute time and memory usage

Here is the plot, anomaly_stats_table.png, showing the table of anomalies, if any, from among all of the checkpoint_runner.sh jobs.

Compute progress

Here is the plot, progress_with_anomalies.png, showing Checkpoint Progress and MaxRSS (SLURM max job memory usage) vs Wall Time.

Execution time and memory usage trends

Here is the plot, resource_trend.png, showing Elapsed (time) and MaxRSS vs. Worker job index.