Changes between Version 11 and Version 12 of Workshops/JobCheckpointing/Examples/BASH


Ignore:
Timestamp:
04/08/2026 02:30:55 PM (3 months ago)
Author:
Carl Baribault
Comment:

Simplified Running section, added first submitter and runner logs

Legend:

Unmodified
Added
Removed
Modified

  • Workshops/JobCheckpointing/Examples/BASH

    v11 v12  
    66Here is an example of a checkpointing application implemented in BASH script.
    77
    8 To run the example, see [wiki://Workshops/JobCheckpointing/Examples/BASH#RunningtheBASHcheckpointingexampleonCypress Running the BASH checkpointing example on Cypress] below.
    9 
    10 [[BR]]
    11 
    12 === checkpoint_signal_iter.sh ===
     8By default, the BASH checkpointing application behaves as follows.
     9
     10 * checkpointing every 20 application iterations (one second per iteration) and
     11 * running a total of 500 iterations.
     12
     13== Running the BASH checkpointing example with defaults ==
     14
     15To run the application on Cypress, perform the following.
     16
     171. Create your own versions of the following files in your current directory on Cypress. (For file editing with '''nano''' on Cypress, see [[https://wiki.hpc.tulane.edu/trac/wiki/cypress/FileEditingSoftware/Example|File Editing Example]].)
     18
     19 * [wiki://Workshops/JobCheckpointing/Examples#checkpoint_submitter.sh checkpoint_submitter.sh]
     20 * [wiki://Workshops/JobCheckpointing/Examples#checkpoint_runner.sh checkpoint_runner.sh]
     21 * [wiki://Workshops/JobCheckpointing/Examples#aggregate_usage.py aggregate_usage.py]
     22 * [wiki://Workshops/JobCheckpointing/Examples#plot_progress.py plot_progress.py]
     23
     242. Create your own version of the following BASH application script file '''checkpoint_signal_iter.sh'''.
    1325
    1426{{{
    … …  
    6779}}}
    6880
    69 == Running the BASH checkpointing example on Cypress ==
    70 
    71 To run the BASH checkpointing job example, defaulting to checkpointing every 20 application iterations and a total of 500 iterations, perform the following.
    72 
    73 1. Create your own versions of the following files in your current directory on Cypress. (For file editing with '''nano''' on Cypress, see [[https://wiki.hpc.tulane.edu/trac/wiki/cypress/FileEditingSoftware/Example|File Editing Example]].)
    74 
    75  * [wiki://Workshops/JobCheckpointing/Examples#checkpoint_submitter.sh checkpoint_submitter.sh]
    76  * [wiki://Workshops/JobCheckpointing/Examples#checkpoint_runner.sh checkpoint_runner.sh]
    77  * [wiki://Workshops/JobCheckpointing/Examples#aggregate_usage.py aggregate_usage.py]
    78  * [wiki://Workshops/JobCheckpointing/Examples#plot_progress.py plot_progress.py]
    79  * [wiki://Workshops/JobCheckpointing/Examples/BASH#checkpoint_signal_iter.sh checkpoint_signal_iter.sh]
    80 
    81 2. Change permissions on the BASH application script, '''checkpoint_signal_iter.sh''' executable via the following command.
     813. Change permissions on the BASH application script, '''checkpoint_signal_iter.sh''', making the script executable via the following '''chmod''' command. (See [wiki://cypress/BasicLinuxComands#chmod chmod].)
    8282
    8383{{{
    … …  
    8585}}}
    8686
    87 3. Submit the job via the following command.
     874. Submit the job via the following command.
    8888
    8989{{{
    9090[tulaneID@cypress1 ~]$ APP_CMD="./checkpoint_signal_iter.sh" \
    9191CKPT_PATH=state_iter_bash.txt \
    92 sbatch checkpoint_runner.sh
    93 }}}
    94 
    95 4. You can monitor the jobs' outputs as they are updated by repeatedly executing the following command, using Ctrl+C to exit files as they finish updating.
    96 
    97 {{{
    98 [tulaneID@cypress1 ~]$ tail -f log_*
     92sbatch checkpoint_submitter.sh
     93}}}
     94
     955. You can monitor job outputs of both the checkpoint_submitter.sh and the checkpoint_runner.sh jobs as they are updated by executing the following '''tail''' command using the '''-F''' option to retry once per second, then using Ctrl+C to exit the retry process entirely.
     96
     97{{{
     98[tulaneID@cypress1 ~]$ tail -F log*
    9999}}}
    100100
    … …  
    143143}}}
    144144
    145 === Listing of the final checkpoint submitter log - completed workflow ===
    146 
    147 The following is the listing of the final log file, '''log_submitter_3303154.out''', generated by '''checkpoint_submitter.sh''' ending with "Workflow complete. Exiting submitter."
     145=== Sample checkpoint submitter logs ===
     146
     147Here are the first and last instances from among the multiple log files generated by the sequence of jobs running the job script [wiki://Workshops/JobCheckpointing/Examples#checkpoint_submitter.sh checkpoint_submitter.sh].
     148
     149The first job in the sequence was submitted by the user, and all other jobs thereafter in that sequence were submitted by the job script itself via '''self-restarting'''.
     150
     151==== Listing of the initial checkpoint submitter log - starting workflow ====
     152
     153Here is the listing of the first log file, '''log_submitter_3303140.out''', generated by [wiki://Workshops/JobCheckpointing/Examples#checkpoint_submitter.sh checkpoint_submitter.sh] ending with the message "Submitter approaching timeout -> restarting self. Submitted batch job 3303142."
     154
     155
     156{{{
     157[tulaneID@cypress2 ~]$cat log_submitter_3303140.out
     158Info[20260401-17:44:05]: Start on cypress01-066; JOBID=3303140
     159Info[20260401-17:44:05]: Submitter settings:
     160Info[20260401-17:44:05]: CKPT_PATH=state_iter_bash.txt
     161Info[20260401-17:44:05]: MAX_ITER=500
     162Info[20260401-17:44:05]: RUNNER_SCRIPT=checkpoint_runner.sh
     163Info[20260401-17:44:05]: SUBMITTER_MARGIN_SEC=60
     164Info[20260401-17:44:05]: SUBMITTER_TIME_LIMIT=3:30
     165Info[20260401-17:44:05]: No previous state found; starting fresh.
     166Info[20260401-17:44:05]: Submitter cycle: 0
     167Info[20260401-17:44:05]: Current checkpoint iteration: 0
     168Info[20260401-17:44:05]: Submitted worker: 3303141
     169Info[20260401-17:44:05]: Waiting on first of worker 3303141 to finish or self-restart.
     170Info[20260401-17:46:46]: Submitter approaching timeout -> restarting self
     171Submitted batch job 3303142
     172}}}
     173
     174==== Listing of the final checkpoint submitter log - completed workflow ====
     175
     176Here is the listing of the final log file, '''log_submitter_3303154.out''', generated by [wiki://Workshops/JobCheckpointing/Examples#checkpoint_submitter.sh checkpoint_submitter.sh] ending with the message "Workflow complete. Exiting submitter."
    148177
    149178{{{
    … …  
    178207}}}
    179208
    180 
    181 === Listing of the final checkpoint runner log - completed application ===
    182 
    183 The following is the listing of the final log file, '''log_runner_3303151.out''', generated by '''checkpoint_runner.sh''' ending with message "Application finished all iterations."
     209=== Sample checkpoint runner logs ===
     210
     211Here are the first and last output logs of the sequence of jobs generated by the job script '''checkpoint_runner.sh'''.
     212
     213All of the jobs in that sequence were submitted by the [wiki://Workshops/JobCheckpointing/Examples#checkpoint_submitter.sh checkpoint_submitter.sh] job script.
     214
     215==== Listing of the initial checkpoint runner log - starting application ====
     216
     217Here is the listing of the intial log file, '''log_runner_3303141.out''', generated by the job script '''checkpoint_runner.sh''' ending with message "Application finished all iterations."
     218
     219{{{
     220[tulaneID@cypress2 ~]$cat log_runner_3303141.out
     221Info[20260401-17:44:05]: Start on cypress01-066; JOBID=3303141
     222Info[20260401-17:44:05]: Runner settings:
     223Info[20260401-17:44:05]: APP_CMD=./checkpoint_signal_iter.sh
     224Info[20260401-17:44:05]: CHECKPOINT_EVERY=20
     225Info[20260401-17:44:05]: CKPT_PATH=state_iter_bash.txt
     226Info[20260401-17:44:05]: LAUNCH_MODE=direct
     227Info[20260401-17:44:05]: MAX_ITER=500
     228Info[20260401-17:44:05]: MODULE_LIST=anaconda3/2023.07
     229Info[20260401-17:44:05]: RUNNER_MARGIN_SEC=60
     230Info[20260401-17:44:05]: RUNNER_TIME_LIMIT=00:03:00
     231Info[20260401-17:44:05]: SRUN_ARGS=-n 1
     232Info[20260401-17:44:05]: Loading modules.
     233Resuming from i=0 (every 20, MAX_ITER=500)
     234[periodic/iter] saved i=20
     235[periodic/iter] saved i=40
     236[periodic/iter] saved i=60
     237[periodic/iter] saved i=80
     238[periodic/iter] saved i=100
     239Bash SIGTERM: saving i=120 and exiting 99
     240Info[20260401-17:46:05]: Program exit code (from timeout wrapper): 124
     241Info[20260401-17:46:05]: Timeout + checkpoint advance detected (0->120).
     242}}}
     243
     244==== Listing of the final checkpoint runner log - completed application ====
     245
     246Here is the listing of the final log file, '''log_runner_3303151.out''', generated by '''checkpoint_runner.sh''' ending with message "Application finished all iterations."
    184247
    185248{{{