Changes between Initial Version and Version 1 of Workshops/JobCheckpointing/Examples/R

Timestamp:

03/14/2026 01:19:36 AM (2 days ago)

Author:

Carl Baribault

Comment:

Initial version, separate page for R checkpointing

Workshops/JobCheckpointing/Examples/R

@@ +1 @@
+[[PageOutline]]
+= R Checkpointing Example =
+
+== Checkpoint Runner ==
+
+See [wiki:Workshops/JobCheckpointing/Examples#CheckpointRunner Checkpoint Runner] for the contents of the job script file '''checkpoint_runner.sh'''.
+
+== R Checkpointing Application ==
+
+=== checkpoint_signal_iter.R ===
+
+{{{
+#!/usr/bin/env Rscript
+# checkpoint_signal_iter.R  (requires 'sigterm' package)
+suppressWarnings(suppressMessages({
+  if (!requireNamespace("sigterm", quietly = TRUE)) {
+    message("ERROR: 'sigterm' package not installed; install via remotes/devtools.")
+    quit(save="no", status=2L)
+  }
+}))
+
+get_env <- function(k, d) { v <- Sys.getenv(k, unset=NA); if (is.na(v) || v=="") d else v }
+CKPT <- get_env("CKPT_PATH", "state_iter.txt")
+EVERY <- as.integer(get_env("CHECKPOINT_EVERY", "20"))
+MAX_ITER <- as.integer(get_env("MAX_ITER", "500"))
+
+atomic_save <- function(path, val) {
+  dir <- dirname(normalizePath(path, mustWork = FALSE))
+  if (!dir.exists(dir)) dir.create(dir, recursive = TRUE, showWarnings = FALSE)
+  tmp <- tempfile(pattern = ".ckpt.", tmpdir = dir)
+  con <- file(tmp, open="wt"); writeLines(as.character(val), con); flush(con); close(con)
+  file.rename(tmp, path)
+}
+load_ckpt <- function(path) {
+  if (!file.exists(path)) return(0L)
+  txt <- tryCatch(readLines(path, warn = FALSE), error = function(e) "0")
+  as.integer(gsub("[^0-9]", "", paste(txt, collapse = "")))
+}
+
+library(sigterm)  # installs a SIGTERM handler; poll has_sigterm_flag()
+
+i <- load_ckpt(CKPT)
+cat(sprintf("R: Resuming from i=%d (every %d, MAX_ITER=%d)\n", i, EVERY, MAX_ITER)); flush(stdout())
+
+repeat {
+  i <- i + 1L
+  Sys.sleep(1)
+
+  if (i %% EVERY == 0L) {
+    atomic_save(CKPT, i)
+    cat(sprintf("[periodic/iter] saved i=%d\n", i)); flush(stdout())
+  }
+
+  if (sigterm::has_sigterm_flag()) {
+    cat(sprintf("R: SIGTERM detected — saving i=%d and exiting 99\n", i)); flush(stdout())
+    atomic_save(CKPT, i)
+    quit(save="no", status=99L)
+  }
+
+  if (i > MAX_ITER) {
+    cat(sprintf("Reached i=%d > %d; exiting 0\n", i, MAX_ITER)); flush(stdout())
+    atomic_save(CKPT, i)
+    quit(save="no", status=0L)
+  }
+}
+}}}
+
+== Running R checkpointing example on Cypress ==
+
+To run the R checkpointing job example, defaulting to checkpointing every 20 application iterations and a total of 500 iterations, perform the following.
+
+1. Edit the files '''checkpoint_runner.sh''' and '''checkpoint_signal_iter.R''' in your current directory.
+ For file editing with nano, etc., see [wiki:cypress/FileEditingSoftware/Example File Editing Example].
+
+2. Use the steps in the section [wiki:cypress/RunningRStudioWithSingularity#Alternativefortidyverse Alternative for tidyverse] for constructing a container image file '''tidyverse_latest.sif''' (or '''tidyverse_4.5.2.sif''' as in the following) for use with Singularity.
+
+3. Install the required R package '''sigterm''' via the following, substituting your own writable R library directory for <R_lib_path>. See [wiki:cypress/InstallingRPackages Installing R Packages on Cypress] for more information on using a writable R library directory.
+
+{{{
+[tulaneID@cypress1 ~]$idev --partition=centos7
+[tulaneID@cypress01-XXX ~]$module load singularity/3.9.0
+[tulaneID@cypress01-XXX ~]$singularity shell tidyverse_4.5.2.sif
+Singularity> Rscript --version # confirm Rscript is available
+Rscript (R) version 4.5.2 (2025-10-31)
+Singularity>Rscript -e "devtools::install_github('atheriel/sigterm', lib='<R_lib_path>')"
+Singularity>exit # exit the container
+[tulaneID@cypress01-XXX ~]$exit # exit the interactive session
+}}}
+
+4. Submit the job via the following command.
+
+{{{
+[tulaneID@cypress1 ~]$APP_CMD="singularity exec tidyverse_4.5.2.sif Rscript checkpoint_signal_iter.R" MODULE_LIST="singularity/3.9.0" CKPT_PATH=state_iter_r.txt sbatch checkpoint_runner.sh
+}}}
+
+5. Monitor the job's output via the following command, substituting the job ID for <jobID>.
+
+{{{
+[tulaneID@cypress1 ~]$ tail -f log_<jobID>.*
+}}}
+
+6. Here are normal results for the output and error files, '''log_<jobID>.err''' and  '''log_<jobID>.out''', observing that the job cancelled and requeued itself many times. (Not all cancellations were captured in the error file.)
+
+{{{
+[tulaneID@cypress1 ~]$cat log_3300700.err
+slurmstepd: *** JOB 3300700 CANCELLED AT 2026-03-13T22:20:20 ***
+slurmstepd: *** JOB 3300700 CANCELLED AT 2026-03-13T22:28:00 ***
+}}}
+ 
+{{{
+[tulaneID@cypress1 ~]$cat log_3300700.out
+Info[20260313-22:18:18]: Start on cypress01-066; JOB_ID=3300700; RESTARTS=0
+Info[20260313-22:18:18]: Settings:
+Info[20260313-22:18:18]: MODULE_LIST=singularity/3.9.0
+Info[20260313-22:18:18]: APP_CMD=singularity exec tidyverse_4.5.2.sif Rscript checkpoint_signal_iter.R
+Info[20260313-22:18:18]: LAUNCH_MODE=direct
+Info[20260313-22:18:18]: SRUN_ARGS=-n 1
+Info[20260313-22:18:18]: TIME_LIMIT=00:03:00
+Info[20260313-22:18:18]: MARGIN_SEC=60
+Info[20260313-22:18:18]: CKPT_PATH=state_iter_r.txt
+Info[20260313-22:18:18]: CHECKPOINT_EVERY=20
+Info[20260313-22:18:18]: MAX_ITER=500
+Info[20260313-22:18:18]: MAX_RESTARTS=10
+=== BEGIN JOB SNAPSHOT (scontrol) ===
+JobId=3300700 Name=ckpt_requeue_demo
+   Priority=80808 Nice=0 Account=<groupID> QOS=normal
+   JobState=RUNNING Reason=None Dependency=(null)
+   Requeue=1 Restarts=0 BatchFlag=1 ExitCode=0:0
+   RunTime=00:00:01 TimeLimit=00:03:00 TimeMin=N/A
+   StartTime=2026-03-13T22:18:18 EndTime=2026-03-13T22:21:18
+   Partition=centos7 AllocNode:Sid=cypress2:33768
+=== END JOB SNAPSHOT (scontrol) ===
+R: Resuming from i=0 (every 20, MAX_ITER=500)
+[periodic/iter] saved i=20
+[periodic/iter] saved i=40
+[periodic/iter] saved i=60
+[periodic/iter] saved i=80
+[periodic/iter] saved i=100
+R: SIGTERM detected — saving i=117 and exiting 99
+Info[20260313-22:20:19]: Program exit code (from timeout wrapper): 124
+Info[20260313-22:20:19]: Timeout TERM observed; checkpoint advanced (0->117). Requeueing...
+Info[20260313-22:20:19]: Requeued via scontrol.
+Info[20260313-22:20:58]: Start on cypress01-066; JOB_ID=3300700; RESTARTS=1
+Info[20260313-22:20:58]: Settings:
+Info[20260313-22:20:58]: MODULE_LIST=singularity/3.9.0
+Info[20260313-22:20:58]: APP_CMD=singularity exec tidyverse_4.5.2.sif Rscript checkpoint_signal_iter.R
+Info[20260313-22:20:58]: LAUNCH_MODE=direct
+Info[20260313-22:20:58]: SRUN_ARGS=-n 1
+Info[20260313-22:20:58]: TIME_LIMIT=00:03:00
+Info[20260313-22:20:58]: MARGIN_SEC=60
+Info[20260313-22:20:58]: CKPT_PATH=state_iter_r.txt
+Info[20260313-22:20:58]: CHECKPOINT_EVERY=20
+Info[20260313-22:20:58]: MAX_ITER=500
+Info[20260313-22:20:58]: MAX_RESTARTS=10
+=== BEGIN JOB SNAPSHOT (scontrol) ===
+JobId=3300700 Name=ckpt_requeue_demo
+   Priority=80808 Nice=0 Account=<groupID> QOS=normal
+   JobState=RUNNING Reason=None Dependency=(null)
+   Requeue=1 Restarts=1 BatchFlag=1 ExitCode=0:0
+   RunTime=00:00:01 TimeLimit=00:03:00 TimeMin=N/A
+   StartTime=2026-03-13T22:20:57 EndTime=2026-03-13T22:23:57
+   Partition=centos7 AllocNode:Sid=cypress2:33768
+=== END JOB SNAPSHOT (scontrol) ===
+R: Resuming from i=117 (every 20, MAX_ITER=500)
+[periodic/iter] saved i=120
+[periodic/iter] saved i=140
+[periodic/iter] saved i=160
+[periodic/iter] saved i=180
+[periodic/iter] saved i=200
+[periodic/iter] saved i=220
+R: SIGTERM detected — saving i=236 and exiting 99
+Info[20260313-22:22:59]: Program exit code (from timeout wrapper): 124
+Info[20260313-22:22:59]: Timeout TERM observed; checkpoint advanced (117->236). Requeueing...
+Info[20260313-22:22:59]: Requeued via scontrol.
+Info[20260313-22:23:29]: Start on cypress01-066; JOB_ID=3300700; RESTARTS=2
+Info[20260313-22:23:29]: Settings:
+Info[20260313-22:23:29]: MODULE_LIST=singularity/3.9.0
+Info[20260313-22:23:29]: APP_CMD=singularity exec tidyverse_4.5.2.sif Rscript checkpoint_signal_iter.R
+Info[20260313-22:23:29]: LAUNCH_MODE=direct
+Info[20260313-22:23:29]: SRUN_ARGS=-n 1
+Info[20260313-22:23:29]: TIME_LIMIT=00:03:00
+Info[20260313-22:23:29]: MARGIN_SEC=60
+Info[20260313-22:23:29]: CKPT_PATH=state_iter_r.txt
+Info[20260313-22:23:29]: CHECKPOINT_EVERY=20
+Info[20260313-22:23:29]: MAX_ITER=500
+Info[20260313-22:23:29]: MAX_RESTARTS=10
+=== BEGIN JOB SNAPSHOT (scontrol) ===
+JobId=3300700 Name=ckpt_requeue_demo
+   Priority=80808 Nice=0 Account=<groupID> QOS=normal
+   JobState=RUNNING Reason=None Dependency=(null)
+   Requeue=1 Restarts=2 BatchFlag=1 ExitCode=0:0
+   RunTime=00:00:02 TimeLimit=00:03:00 TimeMin=N/A
+   StartTime=2026-03-13T22:23:27 EndTime=2026-03-13T22:26:27
+   Partition=centos7 AllocNode:Sid=cypress2:33768
+=== END JOB SNAPSHOT (scontrol) ===
+R: Resuming from i=236 (every 20, MAX_ITER=500)
+[periodic/iter] saved i=240
+[periodic/iter] saved i=260
+[periodic/iter] saved i=280
+[periodic/iter] saved i=300
+[periodic/iter] saved i=320
+[periodic/iter] saved i=340
+R: SIGTERM detected — saving i=355 and exiting 99
+Info[20260313-22:25:30]: Program exit code (from timeout wrapper): 124
+Info[20260313-22:25:30]: Timeout TERM observed; checkpoint advanced (236->355). Requeueing...
+Info[20260313-22:25:30]: Requeued via scontrol.
+Info[20260313-22:25:59]: Start on cypress01-066; JOB_ID=3300700; RESTARTS=3
+Info[20260313-22:25:59]: Settings:
+Info[20260313-22:25:59]: MODULE_LIST=singularity/3.9.0
+Info[20260313-22:25:59]: APP_CMD=singularity exec tidyverse_4.5.2.sif Rscript checkpoint_signal_iter.R
+Info[20260313-22:25:59]: LAUNCH_MODE=direct
+Info[20260313-22:25:59]: SRUN_ARGS=-n 1
+Info[20260313-22:25:59]: TIME_LIMIT=00:03:00
+Info[20260313-22:25:59]: MARGIN_SEC=60
+Info[20260313-22:25:59]: CKPT_PATH=state_iter_r.txt
+Info[20260313-22:25:59]: CHECKPOINT_EVERY=20
+Info[20260313-22:25:59]: MAX_ITER=500
+Info[20260313-22:25:59]: MAX_RESTARTS=10
+=== BEGIN JOB SNAPSHOT (scontrol) ===
+JobId=3300700 Name=ckpt_requeue_demo
+   Priority=80808 Nice=0 Account=<groupID> QOS=normal
+   JobState=RUNNING Reason=None Dependency=(null)
+   Requeue=1 Restarts=3 BatchFlag=1 ExitCode=0:0
+   RunTime=00:00:02 TimeLimit=00:03:00 TimeMin=N/A
+   StartTime=2026-03-13T22:25:57 EndTime=2026-03-13T22:28:57
+   Partition=centos7 AllocNode:Sid=cypress2:33768
+=== END JOB SNAPSHOT (scontrol) ===
+R: Resuming from i=355 (every 20, MAX_ITER=500)
+[periodic/iter] saved i=360
+[periodic/iter] saved i=380
+[periodic/iter] saved i=400
+[periodic/iter] saved i=420
+[periodic/iter] saved i=440
+[periodic/iter] saved i=460
+R: SIGTERM detected — saving i=474 and exiting 99
+Info[20260313-22:28:00]: Program exit code (from timeout wrapper): 124
+Info[20260313-22:28:00]: Timeout TERM observed; checkpoint advanced (355->474). Requeueing...
+Info[20260313-22:28:00]: Requeued via scontrol.
+Info[20260313-22:28:19]: Start on cypress01-066; JOB_ID=3300700; RESTARTS=4
+Info[20260313-22:28:19]: Settings:
+Info[20260313-22:28:19]: MODULE_LIST=singularity/3.9.0
+Info[20260313-22:28:19]: APP_CMD=singularity exec tidyverse_4.5.2.sif Rscript checkpoint_signal_iter.R
+Info[20260313-22:28:19]: LAUNCH_MODE=direct
+Info[20260313-22:28:19]: SRUN_ARGS=-n 1
+Info[20260313-22:28:19]: TIME_LIMIT=00:03:00
+Info[20260313-22:28:19]: MARGIN_SEC=60
+Info[20260313-22:28:19]: CKPT_PATH=state_iter_r.txt
+Info[20260313-22:28:19]: CHECKPOINT_EVERY=20
+Info[20260313-22:28:19]: MAX_ITER=500
+Info[20260313-22:28:19]: MAX_RESTARTS=10
+=== BEGIN JOB SNAPSHOT (scontrol) ===
+JobId=3300700 Name=ckpt_requeue_demo
+   Priority=80808 Nice=0 Account=<groupID> QOS=normal
+   JobState=RUNNING Reason=None Dependency=(null)
+   Requeue=1 Restarts=4 BatchFlag=1 ExitCode=0:0
+   RunTime=00:00:01 TimeLimit=00:03:00 TimeMin=N/A
+   StartTime=2026-03-13T22:28:18 EndTime=2026-03-13T22:31:18
+   Partition=centos7 AllocNode:Sid=cypress2:33768
+=== END JOB SNAPSHOT (scontrol) ===
+R: Resuming from i=474 (every 20, MAX_ITER=500)
+[periodic/iter] saved i=480
+[periodic/iter] saved i=500
+Reached i=501 > 500; exiting 0
+Info[20260313-22:28:47]: Program exit code (from timeout wrapper): 0
+Info[20260313-22:28:47]: Completed.
+}}}