Changes between Version 5 and Version 6 of Workshops/JobCheckpointing/Examples

Timestamp:

04/02/2026 12:41:45 PM (4 months ago)

Author:

Carl Baribault

Comment:

Added headings, references (ongoing)

Workshops/JobCheckpointing/Examples

@@ -2 +2 @@
 = Job Checkpointing Examples =
 
-For a quick start see [wiki:Workshops/JobCheckpointing/Examples#WorkingExamples Working Examples] below.
+'''Before you adopt a checkpointing workflow''', consider whether your workflow can be considered 
+'''embarrassingly parallel''', where the data elements can be processed '''entirely independently''' of each other - rather than each one depending on the previous. (See also [https://www.freecodecamp.org/news/embarrassingly-parallel-algorithms-explained-with-examples/ Embarrassingly Parallel Algorithms Explained].)
+
+If so, namely that your case is the former (embarrassingly parallel) in the above, then please first refer to the workshop [wiki:Workshops/JobParallelism].
+
+== Quick start ==
+
+For a quick start using checkpointing, see [wiki:Workshops/JobCheckpointing/Examples#WorkingExamples Working Examples] below.
+
+== Background on checkpointing ==
 
 Here is an implementation for running '''sequential workflows''' on Cypress - where each component result of the workflow depends the previous component result in the workflow.
@@ -10 +19 @@
 * submitter/runner pattern (see also [https://gjbex.github.io/Workflows-for-HPC/ Workflow for HPC])
 * job chaining (see [https://docs.nersc.gov/jobs/best-practices/ (NERSC) Best Practices for Jobs])
-* sequential workflows
-* restart workflows
+* sequential workflows (see [https://nlsq.readthedocs.io/en/latest/tutorials/routine/three_workflows/hpc_workflow.html Another checkpointing implementation])
+* restart workflows (see [https://compendium.hpc.tu-dresden.de/jobs_and_resources/checkpoint_restart/ Checkpoint Motivation])
 * checkpoint/restart job series
 * using job dependencies to restart jobs that checkpoint their state”
 
-'''But wait''' - before you adopt this type of checkpointing workflow, consider whether your workflow can be considered 
-'''embarrassingly parallel''', where the data elements can be processed '''entirely independently''' of each other - rather than each one depending on the previous. (See also [https://www.freecodecamp.org/news/embarrassingly-parallel-algorithms-explained-with-examples/ Embarrassingly Parallel Algorithms Explained].)
-
-If so, namely that your case is the former (embarrassingly parallel) in the above, then please first refer to the workshop [wiki:Workshops/JobParallelism].
-
-== Checkpoint Submitter ==
+== Checkpoint workflow implementation ==
+
+=== Checkpoint Submitter ===
 
 Here is a job script for a fully self restarting job that controls the workflow by submitting the checkpoint runner job script (see [wiki:Workshops/JobCheckpointing/Examples#CheckpointRunner Checkpoint Runner]) further below.
 
-=== checkpoint_submitter.sh ===
+==== checkpoint_submitter.sh ====
 
 {{{
@@ -222 +228 @@
 }}}
 
-== Checkpoint Runner ==
+=== Checkpoint Runner ===
 
 Here is the checkpoint runner job script for use with the following.
@@ -232 +238 @@
  * For R, see [wiki:Workshops/JobCheckpointing/Examples#RCheckpointingExample R Checkpointint Example]
 
-=== checkpoint_runner.sh ===
+==== checkpoint_runner.sh ====
 
 {{{