Changes between Version 14 and Version 15 of Workshops/JobCheckpointing/Examples

Timestamp:

04/14/2026 11:20:51 AM (3 months ago)

Author:

Carl Baribault

Comment:

Added requeue, violation of acceptable use policy

Workshops/JobCheckpointing/Examples

@@ -6 +6 @@
 This example implements a '''submitter/runner''' (also called cyclic checkpoint/restart) pattern.
 
-A lightweight '''submitter''' job launches a '''runner''' job that executes the '''application''' for a bounded time.
+=== Submitter -> Runner -> Application ===
+
+A lightweight '''submitter''' job launches a '''runner''' job that in turn launches the '''application''' for a bounded time.
 
 After that time, the application writes a checkpoint, and exits cleanly.
 
-The submitter monitors progress and resubmits a new submitter job as needed until the workflow completes. See [wiki://Workshops/JobCheckpointing/Examples#Checkpointworkflowimplementation Checkpoint workflow implementation] below.
+=== Submitter self-resubmits before timing out ===
+
+The submitter itself also runs for a bounded time while monitoring progress. Just before timing out, the submitter resubmits a new submitter job  as needed until the workflow completes using the '''sbatch''' command rather than the SLURM '''requeue''' feature. For details on this checkpointing implementation, see [wiki://Workshops/JobCheckpointing/Examples#Checkpointworkflowimplementation Checkpoint workflow implementation] below.
+
+=== Why do we use sbatch vs SLURM requeue? ===
+
+==== Unacceptable use of requeue ====
+
+The SLURM resource manager provides an option to requeue a running job via the command '''scontrol requeue'''. (See '''man scontrol'''.)
+
+However, when a job is requeued, any previous record of elapsed time used by that same job is '''overwritten'''.
+
+Hence, the requeue option has the following 2 consequences.
+
+ 1. You would '''not''' be able to track the total elapsed time used by your requeued job. 
+ 2. As a result, your job would be in violation of Tulane Information Technology's [https://it.tulane.edu/acceptable-use-policy Acceptable Use Policy], including monitoring your use of Cypress.
 
 == Before you adopt checkpointing ==