Changes between Version 5 and Version 6 of Workshops/JobCheckpointing

Timestamp:

01/22/26 11:08:11 (2 days ago)

Author:

Carl Baribault

Comment:

Added detail to checkpointing process

Workshops/JobCheckpointing

@@ -13 +13 @@
 
 == What is job checkpointing? ==
-Job checkpointing is the process of ensuring or providing for the job's ability to resume processing - without repeating some or all of the processing already performed after termination due to either end of requested walltime or system outage. 
+Job checkpointing is the process of ensuring or programming the job's application software with the ability to save partial results as well as resume processing after termination at end of requested walltime (walltime termination).
 
-== Why is job checkpointing important? ==
+A checkpointed job '''must''' be able to perform the following.
+
+* The application must record it's progress (see the following) at one or both of the following times.
+ * '''At regular time intervals''' on its own (This option is preferred for very long - or sufficiently long - running jobs where system crashes are more likely.) - '''or'''
+ * '''After catching a Signal Terminate signal''' (or '''SIGTERM''') from the Operating System, where the signal is programmed '''in the job script''' to allow for completion of recording before walltime termination. For example, in your job script...
+  * either via sbatch directives
+{{{
+   # --- Append to output and error files ---
+   #SBATCH --open-mode=append
+   # --- Enable automatic requeue ---
+   #SBATCH --requeue
+   # --- Send SIGTERM 2 minutes before walltime ---
+   #SBATCH --signal=TERM@120
+}}}
+  * or bash '''timeout''' command followed by '''requeue'''
+{{{
+timeout 23h ./my_simulation || scontrol requeue $SLURM_JOB_ID
+}}}
+
+* The application must record both the work already performed as well as the current or recent '''state of execution''' or '''state'''.
+
+* When the job is '''requeued''', the application must read the recorded '''state''' and resume from that point of execution with the previous work preserved.
+
+== Why is job checkpointing important - and beneficial? ==
 
 * Checkpointed jobs can get started sooner out of the job queue pending state with a reduced requested run time. (See "backfill scheduling" in [https://slurm.schedmd.com/sched_config.html|SLURM Scheduling Configuration Guide].
-* Most production clusters enforce strict walltime limits. (See see [wiki:cypress/about#SLURMresourcemanager SLURM (resource manager)].)
-* A parallel MPI job can fail as soon as a single node in use crashes.
-* Cloud-based job queues with high availability can enforce the use of pre-emptible job queues.
+* Checkpointed jobs compensate for strict walltime limits enforced by Cypress, LONI, and most other production clusters. (See see [wiki:cypress/about#SLURMresourcemanager SLURM (resource manager)].)
+* Checkpointed jobs running parallel MPI (especially long running jobs recording at regular intervals) can fail as soon as a single node in use crashes.
+* Checkpointed jobs running in certain cloud-based job queues with high availability can experience strictly enforced job pre-emption (SIGTERM signals).
 
-== Impacts of job checkpointing ==
+== What are the impacts of job checkpointing? ==
 
 * Job scripts require provision for requeuing after termination due to timing out or pre-emption.