| 4 | | '''Before you adopt a checkpointing workflow''', consider whether your workflow can be considered |
| 5 | | '''embarrassingly parallel''', where the data elements can be processed '''entirely independently''' of each other - rather than each one depending on the previous. (See also [https://www.freecodecamp.org/news/embarrassingly-parallel-algorithms-explained-with-examples/ Embarrassingly Parallel Algorithms Explained].) |
| 6 | | |
| 7 | | If so, namely that your case is the former (embarrassingly parallel) in the above, then please first refer to the workshop [wiki:Workshops/JobParallelism]. |
| | 4 | == Before you adopt checkpointing == |
| | 5 | |
| | 6 | '''Before you adopt a checkpointing workflow''', make a determination of whether your workflow consists - either entirely or in large part - of data elements or other components that can be processed '''entirely independently''' of each other - rather than each one depending on the previous. This is also referred to as '''embarrassingly parallel'''. (See also [https://www.freecodecamp.org/news/embarrassingly-parallel-algorithms-explained-with-examples/ Embarrassingly Parallel Algorithms Explained].) |
| | 7 | |
| | 8 | If so, namely that your case is the former (embarrassingly parallel) in the above, then please first refer to the material in [wiki:Workshops/JobParallelism Job Parallelism]. |
| 15 | | Here is an implementation for running '''sequential workflows''' on Cypress - where each component result of the workflow depends the previous component result in the workflow. |
| 16 | | |
| 17 | | It can be considered an implementation of an HPC workflow design pattern (see for example [https://github.com/deepakkum21/Books/blob/master/Design%20Patterns%20-%20Elements%20of%20Reusable%20Object%20Oriented%20Software%20-%20GOF.pdf Design Patterns]) that can be described by any of the following expressions. |
| 18 | | |
| 19 | | * submitter/runner pattern (see also [https://gjbex.github.io/Workflows-for-HPC/ Workflow for HPC]) |
| 20 | | * job chaining (see [https://docs.nersc.gov/jobs/best-practices/ (NERSC) Best Practices for Jobs]) |
| 21 | | * sequential workflows (see [https://nlsq.readthedocs.io/en/latest/tutorials/routine/three_workflows/hpc_workflow.html Another checkpointing implementation]) |
| 22 | | * restart workflows (see [https://compendium.hpc.tu-dresden.de/jobs_and_resources/checkpoint_restart/ Checkpoint Motivation]) |
| 23 | | * checkpoint/restart job series |
| 24 | | * using job dependencies to restart jobs that checkpoint their state” |
| | 16 | === Use checkpointing for sequential workflows === |
| | 17 | |
| | 18 | This checkpointing implementation should be used for running '''sequential workflows''' on Cypress - where each component result of the workflow depends the previous component result in the workflow. (See above [wiki:Workshops/JobCheckpointing/Examples#Beforeyouadoptcheckpointing Before you adopt checkpointing].) |
| | 19 | |
| | 20 | === Checkpointing - an HPC workflow design pattern === |
| | 21 | |
| | 22 | This checkpointing implementation can be considered an implementation of an HPC workflow design pattern (see for example [https://github.com/deepakkum21/Books/blob/master/Design%20Patterns%20-%20Elements%20of%20Reusable%20Object%20Oriented%20Software%20-%20GOF.pdf Design Patterns]) that can be described by any of the following expressions. |
| | 23 | |
| | 24 | * submitter/runner pattern (Again see [https://github.com/deepakkum21/Books/blob/master/Design%20Patterns%20-%20Elements%20of%20Reusable%20Object%20Oriented%20Software%20-%20GOF.pdf Design Patterns].) |
| | 25 | * job chaining (See also [https://gjbex.github.io/Workflows-for-HPC/ Workflow for HPC].) |
| | 26 | * sequential workflows (See [https://docs.nersc.gov/jobs/best-practices/ NERSC - Best Practices for Jobs].) |
| | 27 | * restart workflows (See [https://support.ceci-hpc.be/doc/SubmittingJobs/WorkflowManagement/#introduction Wide,Deep, & Cyclic Workflows] and [https://nlsq.readthedocs.io/en/latest/tutorials/routine/three_workflows/hpc_workflow.html Checkpointing implemented in Python].) |
| | 28 | * checkpoint/restart job series (See [https://compendium.hpc.tu-dresden.de/jobs_and_resources/checkpoint_restart/ Checkpoint Motivation].) |
| | 29 | * using job dependencies to restart jobs that checkpoint their state (See [https://github.com/hongthai-lenguyen/Slurm_HPC_Guide/blob/main/slurm_guide.md#52-creating-pipelines-with-job-dependencies Creating Pipelines with (SLURM) Job Dependencies].) |
| | 30 | |
| | 31 | === How does the workflow determine when to stop? === |
| | 32 | |
| | 33 | ==== Limiting the work: using the sbatch --dependencies option ==== |
| | 34 | |
| | 35 | Note that in the context of the last reference in the above list - using job dependencies, when you use the SLURM '''sbatch --dependencies...''' option, the stopping point is not determined automatically. |
| | 36 | |
| | 37 | You may need '''additional effort''' to determine the precise number of '''sbatch --dependencies...''' commands to exiecute - in other words, to determine the length of the dependency chain - before starting the workflow. |
| | 38 | |
| | 39 | ==== Limiting the work: using the application's own work limit ==== |
| | 40 | |
| | 41 | This implementation differs from the above dependency chain, in that rather than using the '''sbatch --dependencies''' option, the stopping point is determined automatically during runtime. |
| | 42 | |
| | 43 | In this implementation, the workflow self restarts or resumes as needed and self terminates when the application's work limit is reached - in other words when the application's work is done. See for example the setting '''MAX_ITER''' in [wiki://Workshops/JobCheckpointing/Examples#checkpoint_submitter.sh checkpoint_submitter.sh] below. |
| | 47 | === Components explained === |
| | 48 | |
| | 49 | In this implementation, the checkpointing workflow consists of the following components. |
| | 50 | |
| | 51 | 1. '''the checkpoint submitter''' - to perform the following |
| | 52 | a. self restart as needed, and |
| | 53 | b. re-submit the runner once per self-restart |
| | 54 | 2. '''the checkpoint runner''' - to perform the following |
| | 55 | a. set its own application timer via the '''timeout''' command |
| | 56 | b. re-start the application as needed |
| | 57 | c. send the signal SIGTERM to the application at timeout |
| | 58 | 3. '''the checkpoint application''' - to perform the following |
| | 59 | a. compute |
| | 60 | b. catch the runner's signal SIGTERM |
| | 61 | c. "handle" the runner's signal SIGTERM by quickly saving the work and stop running (See [wiki:Workshops/JobCheckpointing/Examples#WorkingExamples Working Examples] for applications in BASH, Python and R.) |
| | 62 | |
| 30 | | Here is a job script for a fully self restarting job that controls the workflow by submitting the checkpoint runner job script (see [wiki:Workshops/JobCheckpointing/Examples#CheckpointRunner Checkpoint Runner]) further below. |
| | 65 | Here is the job script for the '''checkpoint submitter''', a fully self-restarting job that controls the workflow by self-restarting as many times as needed and submitting one instance per each restart of the checkpoint runner job script. (See [wiki:Workshops/JobCheckpointing/Examples#CheckpointRunner Checkpoint Runner] further below.) |
| | 66 | |
| | 67 | ==== Overriding default settings ==== |
| | 68 | |
| | 69 | This checkpoint submitter uses settings - with default values - implemented using environment variables (see [wiki:cypress/WorkingOnClusters#EnvironmentVariables Environment Variables]), where the default values can be overridden by either of the following. |
| | 70 | |
| | 71 | 1. Override default settings by one or more '''export''' commands prior to the '''sbatch''' command |
| | 72 | |
| | 73 | {{{ |
| | 74 | # example: export all on one line, then submit |
| | 75 | [tulaneId@cypress1 ~]$export VAR1=<value1> VAR2=<value2> ... |
| | 76 | [tulaneId@cypress1 ~]$sbatch checkpoint_submitter.sh |
| | 77 | }}} |
| | 78 | |
| | 79 | {{{ |
| | 80 | # example: or export one line for each variable, then submit |
| | 81 | [tulaneId@cypress1 ~]$export VAR1=<value1> |
| | 82 | [tulaneId@cypress1 ~]$export VAR2=<value2> |
| | 83 | ... |
| | 84 | [tulaneId@cypress1 ~]$sbatch checkpoint_submitter.sh |
| | 85 | }}} |
| | 86 | |
| | 87 | 2. Alternatively, override settings by assigning values at the beginning, in front of the '''sbatch''' command |
| | 88 | |
| | 89 | {{{ |
| | 90 | # example: all on one line with submit |
| | 91 | [tulaneId@cypress1 ~]$VAR1=<value1> VAR2=<value2> ... sbatch checkpoint_submitter.sh |
| | 92 | }}} |
| | 93 | |
| | 94 | {{{ |
| | 95 | # example: on multiple lines using the line continuation character "\" with submit |
| | 96 | [tulaneId@cypress1 ~]$VAR1=<value1> \ |
| | 97 | VAR2=<value2> \ |
| | 98 | ... \ |
| | 99 | sbatch checkpoint_submitter.sh |
| | 100 | }}} |
