[[PageOutline]] = R Checkpointing Example = Here is an example of a checkpointing application implemented in R script. By default, the R checkpointing application behaves as follows. * checkpointing every 20 application iterations (one second per iteration) and * running a total of 500 iterations. == Running the R checkpointing example with defaults == To run the application on Cypress, perform the following. 1. Create your own versions of the following files in your current directory on Cypress. (For file editing with '''nano''' on Cypress, see [[https://wiki.hpc.tulane.edu/trac/wiki/cypress/FileEditingSoftware/Example|File Editing Example]].) * [wiki://Workshops/JobCheckpointing/Examples#checkpoint_submitter.sh checkpoint_submitter.sh] * [wiki://Workshops/JobCheckpointing/Examples#checkpoint_runner.sh checkpoint_runner.sh] * [wiki://Workshops/JobCheckpointing/Examples#aggregate_usage.py aggregate_usage.py] * [wiki://Workshops/JobCheckpointing/Examples#plot_progress.py plot_progress.py] 2. Create your own version of the following R application script file '''checkpoint_signal_R.sh'''. {{{ #!/usr/bin/env Rscript # checkpoint_signal_iter.R (requires 'sigterm' package) suppressWarnings(suppressMessages({ if (!requireNamespace("sigterm", quietly = TRUE)) { message("ERROR: 'sigterm' package not installed; install via remotes/devtools.") quit(save="no", status=2L) } })) get_env <- function(k, d) { v <- Sys.getenv(k, unset=NA); if (is.na(v) || v=="") d else v } CKPT <- get_env("CKPT_PATH", "state_iter.txt") EVERY <- as.integer(get_env("CHECKPOINT_EVERY", "20")) MAX_ITER <- as.integer(get_env("MAX_ITER", "500")) atomic_save <- function(path, val) { dir <- dirname(normalizePath(path, mustWork = FALSE)) if (!dir.exists(dir)) dir.create(dir, recursive = TRUE, showWarnings = FALSE) tmp <- tempfile(pattern = ".ckpt.", tmpdir = dir) con <- file(tmp, open="wt"); writeLines(as.character(val), con); flush(con); close(con) file.rename(tmp, path) } load_ckpt <- function(path) { if (!file.exists(path)) return(0L) txt <- tryCatch(readLines(path, warn = FALSE), error = function(e) "0") as.integer(gsub("[^0-9]", "", paste(txt, collapse = ""))) } library(sigterm) # installs a SIGTERM handler; poll has_sigterm_flag() i <- load_ckpt(CKPT) cat(sprintf("R: Resuming from i=%d (every %d, MAX_ITER=%d)\n", i, EVERY, MAX_ITER)); flush(stdout()) repeat { i <- i + 1L Sys.sleep(1) if (i %% EVERY == 0L) { atomic_save(CKPT, i) cat(sprintf("[periodic/iter] saved i=%d\n", i)); flush(stdout()) } if (sigterm::has_sigterm_flag()) { cat(sprintf("R: SIGTERM detected — saving i=%d and exiting 99\n", i)); flush(stdout()) atomic_save(CKPT, i) quit(save="no", status=99L) } if (i > MAX_ITER) { cat(sprintf("Reached i=%d > %d; exiting 0\n", i, MAX_ITER)); flush(stdout()) atomic_save(CKPT, i) quit(save="no", status=0L) } } }}} 3. Use the steps in the section [wiki:cypress/RunningRStudioWithSingularity#Alternativefortidyverse Alternative for tidyverse] for constructing a container image file '''tidyverse_latest.sif''' (or '''tidyverse_4.5.2.sif''' as in the following) for use with Singularity. 4. Install the required R package '''sigterm''' via the following, substituting your own writable R library directory for . See [wiki:cypress/InstallingRPackages Installing R Packages on Cypress] for more information on using a writable R library directory. {{{ [tulaneID@cypress1 ~]$idev --partition=centos7 [tulaneID@cypress01-XXX ~]$module load singularity/3.9.0 [tulaneID@cypress01-XXX ~]$singularity shell tidyverse_4.5.2.sif Singularity> Rscript --version # confirm Rscript is available Rscript (R) version 4.5.2 (2025-10-31) Singularity>Rscript -e "devtools::install_github('atheriel/sigterm', lib='')" Singularity>exit # exit the container [tulaneID@cypress01-XXX ~]$exit # exit the interactive session }}} 5. Submit the job via the following command. {{{ [tulaneID@cypress1 ~]$APP_CMD="singularity exec tidyverse_4.5.2.sif Rscript checkpoint_signal_iter.R" \ MODULE_LIST="singularity/3.9.0" \ CKPT_PATH=state_iter_r.txt \ sbatch checkpoint_submitter.sh }}} 6. You can monitor job outputs of both the checkpoint_submitter.sh and the checkpoint_runner.sh jobs as they are updated by executing the following '''tail''' command using the '''-F''' option to retry once per second, then using Ctrl+C to exit the retry process entirely. {{{ [tulaneID@cypress1 ~]$ tail -F log* }}} == Overriding the default settings == To customize your workflow, see [wiki://Workshops/JobCheckpointing/Examples#Overridingdefaultsettings Overriding default settings] for overriding the default settings in the checkpoint submitter according to your needs. == Sample run outputs == === List of output files === Here is a list of all output files from a sample run sorted by time, latest-to-earliest {{{ [tulaneID@cypress2 ~]$ls -lt *.csv *.err *.out *.png *.txt -rw-r--r-- 1 tulaneID hpcstaff 52539 Apr 1 17:07 anomaly_stats_table.png -rw-r--r-- 1 tulaneID hpcstaff 1399 Apr 1 17:07 log_submitter_3303072.out -rw-r--r-- 1 tulaneID hpcstaff 468 Apr 1 17:07 anomaly_details.csv -rw-r--r-- 1 tulaneID hpcstaff 635 Apr 1 17:07 log_submitter_3303072.err -rw-r--r-- 1 tulaneID hpcstaff 52642 Apr 1 17:07 progress_with_anomalies.png -rw-r--r-- 1 tulaneID hpcstaff 41081 Apr 1 17:07 resource_trend.png -rw-r--r-- 1 tulaneID hpcstaff 510 Apr 1 17:07 worker_usage.csv -rw-r--r-- 1 tulaneID hpcstaff 722 Apr 1 17:07 log_submitter_3303070.out -rw-r--r-- 1 tulaneID hpcstaff 18 Apr 1 17:07 submitter_state.txt -rw-r--r-- 1 tulaneID hpcstaff 888 Apr 1 17:04 log_runner_3303071.out -rw-r--r-- 1 tulaneID hpcstaff 4 Apr 1 17:04 state_iter_r.txt -rw-r--r-- 1 tulaneID hpcstaff 0 Apr 1 17:04 log_runner_3303071.err -rw-r--r-- 1 tulaneID hpcstaff 722 Apr 1 17:04 log_submitter_3303067.out -rw-r--r-- 1 tulaneID hpcstaff 0 Apr 1 17:04 log_submitter_3303070.err -rw-r--r-- 1 tulaneID hpcstaff 40 Apr 1 17:04 worker_ids.txt -rw-r--r-- 1 tulaneID hpcstaff 1036 Apr 1 17:03 log_runner_3303068.out -rw-r--r-- 1 tulaneID hpcstaff 0 Apr 1 17:01 log_runner_3303068.err -rw-r--r-- 1 tulaneID hpcstaff 722 Apr 1 17:01 log_submitter_3303064.out -rw-r--r-- 1 tulaneID hpcstaff 0 Apr 1 17:01 log_submitter_3303067.err -rw-r--r-- 1 tulaneID hpcstaff 1036 Apr 1 17:01 log_runner_3303065.out -rw-r--r-- 1 tulaneID hpcstaff 0 Apr 1 16:59 log_runner_3303065.err -rw-r--r-- 1 tulaneID hpcstaff 722 Apr 1 16:59 log_submitter_3303062.out -rw-r--r-- 1 tulaneID hpcstaff 0 Apr 1 16:59 log_submitter_3303064.err -rw-r--r-- 1 tulaneID hpcstaff 1036 Apr 1 16:58 log_runner_3303063.out -rw-r--r-- 1 tulaneID hpcstaff 0 Apr 1 16:56 log_runner_3303063.err -rw-r--r-- 1 tulaneID hpcstaff 729 Apr 1 16:56 log_submitter_3303060.out -rw-r--r-- 1 tulaneID hpcstaff 0 Apr 1 16:56 log_submitter_3303062.err -rw-r--r-- 1 tulaneID hpcstaff 1000 Apr 1 16:56 log_runner_3303061.out -rw-r--r-- 1 tulaneID hpcstaff 0 Apr 1 16:54 log_runner_3303061.err -rw-r--r-- 1 tulaneID hpcstaff 0 Apr 1 16:54 log_submitter_3303060.err }}} === Sample checkpoint submitter logs === Here are the first and last instances from among the multiple log files generated by the sequence of jobs running the job script [wiki://Workshops/JobCheckpointing/Examples#checkpoint_submitter.sh checkpoint_submitter.sh]. The first job in the sequence was submitted by the user, and all other jobs thereafter in that sequence were submitted by the job script itself via '''self-restarting'''. ==== Listing of the initial checkpoint submitter log - starting workflow ==== Here is the listing of the first log file, '''log_submitter_3303060.out''', generated by [wiki://Workshops/JobCheckpointing/Examples#checkpoint_submitter.sh checkpoint_submitter.sh] ending with the message "Submitter approaching timeout -> restarting self. Submitted batch job 3303142." {{{ [tulaneID@cypress2 ~]$cat log_submitter_3303060.out Info[20260401-16:54:06]: Start on cypress01-066; JOBID=3303060 Info[20260401-16:54:06]: Submitter settings: Info[20260401-16:54:06]: CKPT_PATH=state_iter_r.txt Info[20260401-16:54:06]: MAX_ITER=500 Info[20260401-16:54:06]: RUNNER_SCRIPT=checkpoint_runner.sh Info[20260401-16:54:06]: SUBMITTER_MARGIN_SEC=60 Info[20260401-16:54:06]: SUBMITTER_TIME_LIMIT=3:30 Info[20260401-16:54:06]: No previous state found; starting fresh. Info[20260401-16:54:06]: Submitter cycle: 0 Info[20260401-16:54:06]: Current checkpoint iteration: 0 Info[20260401-16:54:06]: Submitted worker: 3303061 Info[20260401-16:54:06]: Waiting on worker 3303061 Info[20260401-16:56:46]: Submitter approaching timeout -> restarting self Submitted batch job 3303062 }}} ==== Listing of the final checkpoint submitter log - completed workflow ==== Here is the listing of the final log file, '''log_submitter_3303072.out''', generated by [wiki://Workshops/JobCheckpointing/Examples#checkpoint_submitter.sh checkpoint_submitter.sh] ending with the message "Workflow complete. Exiting submitter." {{{ [tulaneID@cypress2 ~]$cat log_submitter_3303072.out Info[20260401-17:07:08]: Start on cypress01-066; JOBID=3303072 Info[20260401-17:07:08]: Submitter settings: Info[20260401-17:07:08]: CKPT_PATH=state_iter_r.txt Info[20260401-17:07:08]: MAX_ITER=500 Info[20260401-17:07:08]: RUNNER_SCRIPT=checkpoint_runner.sh Info[20260401-17:07:08]: SUBMITTER_MARGIN_SEC=60 Info[20260401-17:07:08]: SUBMITTER_TIME_LIMIT=3:30 Info[20260401-17:07:08]: Loaded state: submitter_cycle=5 Info[20260401-17:07:08]: Submitter cycle: 5 Info[20260401-17:07:08]: Current checkpoint iteration: 501 Info[20260401-17:07:08]: Reached MAX_ITER=500; running final analytics. Info[20260401-17:07:08]: Running analytics aggregate_usage.py ... Wrote worker_usage.csv JobID ElapsedSeconds TotalCPU ... ReqMem NodeList State 0 3303061 121 00:03.487 ... 512Mn cypress01-066 FAILED 1 3303063 121 00:03.506 ... 512Mn cypress01-066 FAILED 2 3303065 121 00:03.502 ... 512Mn cypress01-066 FAILED 3 3303068 121 00:03.484 ... 512Mn cypress01-066 FAILED 4 3303071 27 00:03.478 ... 512Mn cypress01-066 COMPLETED [5 rows x 9 columns] Info[20260401-17:07:10]: Running analytics plot_progress.py ... Saved progress_with_anomalies.png Saved resource_trend.png Saved anomaly_details.csv Saved anomaly_stats_table.png Info[20260401-17:07:14]: Workflow complete. Exiting submitter. }}} === Sample checkpoint runner logs === Here are the first and last output logs of the sequence of jobs generated by the job script [wiki://Workshops/JobCheckpointing/Examples#checkpoint_runner.sh checkpoint_runner.sh]. All of the jobs in that sequence were submitted by the [wiki://Workshops/JobCheckpointing/Examples#checkpoint_submitter.sh checkpoint_submitter.sh] job script. ==== Listing of the initial checkpoint runner log - starting application ==== Here is the listing of the intial log file, '''log_runner_3303061.out''', generated by the job script [wiki://Workshops/JobCheckpointing/Examples#checkpoint_runner.sh checkpoint_runner.sh] ending with message "Timeout + checkpoint advance detected (0->120)." {{{ [tulaneID@cypress2 ~]$cat log_runner_3303061.out Info[20260401-16:54:06]: Start on cypress01-066; JOBID=3303061 Info[20260401-16:54:06]: Runner settings: Info[20260401-16:54:06]: APP_CMD=singularity exec tidyverse_4.5.2.sif Rscript checkpoint_signal_iter.R Info[20260401-16:54:06]: CHECKPOINT_EVERY=20 Info[20260401-16:54:06]: CKPT_PATH=state_iter_r.txt Info[20260401-16:54:06]: LAUNCH_MODE=direct Info[20260401-16:54:06]: MAX_ITER=500 Info[20260401-16:54:06]: MODULE_LIST=singularity/3.9.0 Info[20260401-16:54:06]: RUNNER_MARGIN_SEC=60 Info[20260401-16:54:06]: RUNNER_TIME_LIMIT=00:03:00 Info[20260401-16:54:06]: SRUN_ARGS=-n 1 Info[20260401-16:54:06]: Loading modules. R: Resuming from i=0 (every 20, MAX_ITER=500) [periodic/iter] saved i=20 [periodic/iter] saved i=40 [periodic/iter] saved i=60 [periodic/iter] saved i=80 [periodic/iter] saved i=100 R: SIGTERM detected — saving i=119 and exiting 99 Info[20260401-16:56:07]: Program exit code (from timeout wrapper): 124 Info[20260401-16:56:07]: Timeout + checkpoint advance detected (0->119). }}} ==== Listing of the final checkpoint runner log - completed application ==== Here is the listing of the final log file, '''log_runner_3303171.out''', generated by [wiki://Workshops/JobCheckpointing/Examples#checkpoint_runner.sh checkpoint_runner.sh] ending with message "Application finished all iterations." {{{ [tulaneID@cypress2 ~]$cat log_runner_3303071.out Info[20260401-17:04:28]: Start on cypress01-066; JOBID=3303071 Info[20260401-17:04:28]: Runner settings: Info[20260401-17:04:28]: APP_CMD=singularity exec tidyverse_4.5.2.sif Rscript checkpoint_signal_iter.R Info[20260401-17:04:28]: CHECKPOINT_EVERY=20 Info[20260401-17:04:28]: CKPT_PATH=state_iter_r.txt Info[20260401-17:04:28]: LAUNCH_MODE=direct Info[20260401-17:04:28]: MAX_ITER=500 Info[20260401-17:04:28]: MODULE_LIST=singularity/3.9.0 Info[20260401-17:04:28]: RUNNER_MARGIN_SEC=60 Info[20260401-17:04:28]: RUNNER_TIME_LIMIT=00:03:00 Info[20260401-17:04:28]: SRUN_ARGS=-n 1 Info[20260401-17:04:28]: Loading modules. R: Resuming from i=476 (every 20, MAX_ITER=500) [periodic/iter] saved i=480 [periodic/iter] saved i=500 Reached i=501 > 500; exiting 0 Info[20260401-17:04:55]: Program exit code (from timeout wrapper): 0 Info[20260401-17:04:55]: Application finished all iterations. }}} === Listings of post-workflow analysis output files === The following is the complete, annotated listing of all the generated .csv and .txt files {{{ [tulaneID@cypress2 ~]$tail -n +1 *.csv *.txt ==> anomaly_details.csv <== JobID,StartTime,ElapsedSeconds,MaxRSS_MiB,TimeZ,MemModZ,AnomalyType 3303061,2026-04-01 16:54:06,121.0,91.64047241210938,0.4999999999999999,0.0,none 3303063,2026-04-01 16:56:46,121.0,93.33038330078124,0.4999999999999999,0.0,none 3303065,2026-04-01 16:59:17,121.0,91.4154052734375,0.4999999999999999,0.0,none 3303068,2026-04-01 17:01:57,121.0,95.40939331054688,0.4999999999999999,0.0,none 3303071,2026-04-01 17:04:28,27.0,44.5709228515625,-2.0,10.857998043052838,memory ==> worker_usage.csv <== JobID,ElapsedSeconds,TotalCPU,MaxRSS,MaxRSS_bytes,MaxRSS_MiB,ReqMem,NodeList,State 3303061,121,00:03.487,96092K,96092000.0,91.64047241210938,512Mn,cypress01-066,FAILED 3303063,121,00:03.506,97864K,97864000.0,93.33038330078125,512Mn,cypress01-066,FAILED 3303065,121,00:03.502,95856K,95856000.0,91.4154052734375,512Mn,cypress01-066,FAILED 3303068,121,00:03.484,100044K,100044000.0,95.40939331054688,512Mn,cypress01-066,FAILED 3303071,27,00:03.478,46736K,46736000.0,44.5709228515625,512Mn,cypress01-066,COMPLETED ==> state_iter_r.txt <== 501 ==> submitter_state.txt <== submitter_cycle=5 ==> worker_ids.txt <== 3303061 3303063 3303065 3303068 3303071 }}} === Post-workflow analysis plots === ==== Background ==== For background on the following plots, see [wiki://Workshops/JobCheckpointing/Examples#Plottingexecutiontimeandmemoryusage Plotting execution time and memory usage]. ==== Table of anomalies, compute time and memory usage ==== Here is the plot, '''anomaly_stats_table.png''', showing the table of anomalies, if any, from among all of the [wiki://Workshops/JobCheckpointing/Examples#checkpoint_runner.sh checkpoint_runner.sh] jobs. [[Image(R_anomaly_stats_table.png, width=800px)]] ==== Compute progress ==== Here is the plot, '''progress_with_anomalies.png''', showing '''Checkpoint Progress''' and '''MaxRSS''' (SLURM max job memory usage) vs '''Wall Time'''. [[Image(R_progress_with_anomalies.png, width=800px)]] ==== Execution time and memory usage trends ==== Here is the plot, '''resource_trend.png''', showing '''Elapsed (time)''' and '''MaxRSS''' vs. '''Worker job index'''. [[Image(R_resource_trend.png, width=800px)]]