Changes between Version 3 and Version 4 of Workshops/JobCheckpointing/Examples/Python


Ignore:
Timestamp:
04/08/2026 04:00:58 PM (3 months ago)
Author:
Carl Baribault
Comment:

Updated for checkpoint_submitter.sh

Legend:

Unmodified
Added
Removed
Modified

  • Workshops/JobCheckpointing/Examples/Python

    v3 v4  
    11[[PageOutline]]
    2 = Python Checkpointing Example - Under construction =
    3 
    4 '''This page is under construction to comply with the checkpoint submitter/runner implementation.'''
    5 
    6 == Checkpoint Runner ==
    7 
    8 See [wiki:Workshops/JobCheckpointing/Examples#CheckpointRunner Checkpoint Runner] for the contents of the job script file '''checkpoint_runner.sh'''.
     2= Python Checkpointing Example =
    93
    104== Python Checkpointing Application ==
    115
    12 === checkpoint_signal_iter.py ===
     6Here is an example of a checkpointing application implemented in Python script.
     7
     8By default, the Python checkpointing application behaves as follows.
     9
     10 * checkpointing every 20 application iterations (one second per iteration) and
     11 * running a total of 500 iterations.
     12
     13== Running the Python checkpointing example with defaults ==
     14
     15To run the application on Cypress, perform the following.
     16
     171. Create your own versions of the following files in your current directory on Cypress. (For file editing with '''nano''' on Cypress, see [[https://wiki.hpc.tulane.edu/trac/wiki/cypress/FileEditingSoftware/Example|File Editing Example]].)
     18
     19 * [wiki://Workshops/JobCheckpointing/Examples#checkpoint_submitter.sh checkpoint_submitter.sh]
     20 * [wiki://Workshops/JobCheckpointing/Examples#checkpoint_runner.sh checkpoint_runner.sh]
     21 * [wiki://Workshops/JobCheckpointing/Examples#aggregate_usage.py aggregate_usage.py]
     22 * [wiki://Workshops/JobCheckpointing/Examples#plot_progress.py plot_progress.py]
     23
     242. Create your own version of the following Python application script file '''checkpoint_signal_iter.py'''.
    1325
    1426{{{
    … …  
    7688}}}
    7789
    78 == Running Python checkpointing example on Cypress ==
    79 
    80 To run the Python checkpointing job example, defaulting to checkpointing every 20 application iterations and a total of 500 iterations, perform the following.
    81 
    82 1. Edit the files '''checkpoint_runner.sh''' and '''checkpoint_signal_iter.py''' as shown above in your current directory.
    83  For file editing with nano, etc., see [[https://wiki.hpc.tulane.edu/trac/wiki/cypress/FileEditingSoftware/Example|File Editing Example]].
    84 
    85 2. Submit the job via the following command.
     903. Submit the job via the following command.
    8691
    8792{{{
    … …  
    8994}}}
    9095
    91 3. Monitor the job's output via the following command, substituting the job ID for <jobID>.
    92 
    93 {{{
    94 [tulaneID@cypress1 ~]$ tail -f log_<jobID>.*
    95 }}}
    96 
    97 4. Here are normal results for the output and error files, '''log_<jobID>.err''' and  '''log_<jobID>.out''', observing that the job cancelled and requeued itself many times. (Not all cancellations were captured in the error file.)
    98 
    99 {{{
    100 [tulaneID@cypress1 ~]$cat log_3300699.err
    101 slurmstepd: *** JOB 3300699 CANCELLED AT 2026-03-13T22:22:57 ***
    102 slurmstepd: *** JOB 3300699 CANCELLED AT 2026-03-13T22:25:27 ***
    103 slurmstepd: *** JOB 3300699 CANCELLED AT 2026-03-13T22:27:57 ***
    104 }}}
    105  
    106 {{{
    107 [tulaneID@cypress1 ~]$cat log_3300699.out
    108 Info[20260313-22:18:16]: Start on cypress01-066; JOB_ID=3300699; RESTARTS=0
    109 Info[20260313-22:18:16]: Settings:
    110 Info[20260313-22:18:16]: MODULE_LIST=anaconda3/2023.07
    111 Info[20260313-22:18:16]: APP_CMD=python3 checkpoint_signal_iter.py
    112 Info[20260313-22:18:16]: LAUNCH_MODE=direct
    113 Info[20260313-22:18:16]: SRUN_ARGS=-n 1
    114 Info[20260313-22:18:16]: TIME_LIMIT=00:03:00
    115 Info[20260313-22:18:16]: MARGIN_SEC=60
    116 Info[20260313-22:18:16]: CKPT_PATH=state_iter_py.txt
    117 Info[20260313-22:18:16]: CHECKPOINT_EVERY=20
    118 Info[20260313-22:18:16]: MAX_ITER=500
    119 Info[20260313-22:18:16]: MAX_RESTARTS=10
    120 === BEGIN JOB SNAPSHOT (scontrol) ===
    121 JobId=3300699 Name=ckpt_requeue_demo
    122    Priority=80808 Nice=0 Account=<groupID> QOS=normal
    123    JobState=RUNNING Reason=None Dependency=(null)
    124    Requeue=1 Restarts=0 BatchFlag=1 ExitCode=0:0
    125    RunTime=00:00:01 TimeLimit=00:03:00 TimeMin=N/A
    126    StartTime=2026-03-13T22:18:16 EndTime=2026-03-13T22:21:16
    127    Partition=centos7 AllocNode:Sid=cypress2:33768
    128 === END JOB SNAPSHOT (scontrol) ===
     964. You can monitor job outputs of both the checkpoint_submitter.sh and the checkpoint_runner.sh jobs as they are updated by executing the following '''tail''' command using the '''-F''' option to retry once per second, then using Ctrl+C to exit the retry process entirely.
     97
     98{{{
     99[tulaneID@cypress1 ~]$ tail -F log*
     100}}}
     101
     102== Overriding the default settings ==
     103
     104To customize your workflow, see [wiki://Workshops/JobCheckpointing/Examples#Overridingdefaultsettings Overriding default settings] for overriding the default settings in the checkpoint submitter according to your needs.
     105
     106== Sample run outputs ==
     107
     108=== List of output files ===
     109
     110Here is a list of all output files from a sample run sorted by time, latest-to-earliest
     111
     112{{{
     113[tulaneID@cypress2 ~]$ls -lt *.csv *.err *.out *.png *.txt
     114-rw-r--r-- 1 tulaneID hpcstaff  1404 Apr  1 16:14 log_submitter_3303041.out
     115-rw-r--r-- 1 tulaneID hpcstaff   480 Apr  1 16:14 anomaly_details.csv
     116-rw-r--r-- 1 tulaneID hpcstaff 54778 Apr  1 16:14 anomaly_stats_table.png
     117-rw-r--r-- 1 tulaneID hpcstaff 44362 Apr  1 16:14 resource_trend.png
     118-rw-r--r-- 1 tulaneID hpcstaff   757 Apr  1 16:14 log_submitter_3303041.err
     119-rw-r--r-- 1 tulaneID hpcstaff 48683 Apr  1 16:14 progress_with_anomalies.png
     120-rw-r--r-- 1 tulaneID hpcstaff   509 Apr  1 16:14 worker_usage.csv
     121-rw-r--r-- 1 tulaneID hpcstaff   727 Apr  1 16:14 log_submitter_3303039.out
     122-rw-r--r-- 1 tulaneID hpcstaff    18 Apr  1 16:14 submitter_state.txt
     123-rw-r--r-- 1 tulaneID hpcstaff   824 Apr  1 16:12 log_runner_3303040.out
     124-rw------- 1 tulaneID hpcstaff     3 Apr  1 16:12 state_iter_py.txt
     125-rw-r--r-- 1 tulaneID hpcstaff     0 Apr  1 16:11 log_runner_3303040.err
     126-rw-r--r-- 1 tulaneID hpcstaff   727 Apr  1 16:11 log_submitter_3303037.out
     127-rw-r--r-- 1 tulaneID hpcstaff     0 Apr  1 16:11 log_submitter_3303039.err
     128-rw-r--r-- 1 tulaneID hpcstaff    40 Apr  1 16:11 worker_ids.txt
     129-rw-r--r-- 1 tulaneID hpcstaff   957 Apr  1 16:11 log_runner_3303038.out
     130-rw-r--r-- 1 tulaneID hpcstaff     0 Apr  1 16:09 log_runner_3303038.err
     131-rw-r--r-- 1 tulaneID hpcstaff   727 Apr  1 16:09 log_submitter_3303035.out
     132-rw-r--r-- 1 tulaneID hpcstaff     0 Apr  1 16:09 log_submitter_3303037.err
     133-rw-r--r-- 1 tulaneID hpcstaff   957 Apr  1 16:08 log_runner_3303036.out
     134-rw-r--r-- 1 tulaneID hpcstaff     0 Apr  1 16:06 log_runner_3303036.err
     135-rw-r--r-- 1 tulaneID hpcstaff   727 Apr  1 16:06 log_submitter_3303033.out
     136-rw-r--r-- 1 tulaneID hpcstaff     0 Apr  1 16:06 log_submitter_3303035.err
     137-rw-r--r-- 1 tulaneID hpcstaff   957 Apr  1 16:05 log_runner_3303034.out
     138-rw-r--r-- 1 tulaneID hpcstaff     0 Apr  1 16:03 log_runner_3303034.err
     139-rw-r--r-- 1 tulaneID hpcstaff   734 Apr  1 16:03 log_submitter_3303031.out
     140-rw-r--r-- 1 tulaneID hpcstaff     0 Apr  1 16:03 log_submitter_3303033.err
     141-rw-r--r-- 1 tulaneID hpcstaff   949 Apr  1 16:03 log_runner_3303032.out
     142-rw-r--r-- 1 tulaneID hpcstaff     0 Apr  1 16:01 log_runner_3303032.err
     143-rw-r--r-- 1 tulaneID hpcstaff     0 Apr  1 16:01 log_submitter_3303031.err
     144}}}
     145
     146=== Sample checkpoint submitter logs ===
     147
     148Here are the first and last instances from among the multiple log files generated by the sequence of jobs running the job script [wiki://Workshops/JobCheckpointing/Examples#checkpoint_submitter.sh checkpoint_submitter.sh].
     149
     150The first job in the sequence was submitted by the user, and all other jobs thereafter in that sequence were submitted by the job script itself via '''self-restarting'''.
     151
     152==== Listing of the initial checkpoint submitter log - starting workflow ====
     153
     154Here is the listing of the first log file, '''log_submitter_3303031.out''', generated by [wiki://Workshops/JobCheckpointing/Examples#checkpoint_submitter.sh checkpoint_submitter.sh] ending with the message "Submitter approaching timeout -> restarting self. Submitted batch job 3303033."
     155
     156{{{
     157[tulaneID@cypress2 ~]$cat log_submitter_3303031.out
     158Info[20260401-16:01:12]: Start on cypress01-066; JOBID=3303031
     159Info[20260401-16:01:12]: Submitter settings:
     160Info[20260401-16:01:12]: CKPT_PATH=state_iter_py.txt
     161Info[20260401-16:01:12]: MAX_ITER=500
     162Info[20260401-16:01:12]: RUNNER_SCRIPT=checkpoint_runner.sh
     163Info[20260401-16:01:12]: SUBMITTER_MARGIN_SEC=60
     164Info[20260401-16:01:12]: SUBMITTER_TIME_LIMIT=00:03:30
     165Info[20260401-16:01:12]: No previous state found; starting fresh.
     166Info[20260401-16:01:12]: Submitter cycle: 0
     167Info[20260401-16:01:12]: Current checkpoint iteration: 0
     168Info[20260401-16:01:12]: Submitted worker: 3303032
     169Info[20260401-16:01:12]: Waiting on worker 3303032
     170Info[20260401-16:03:52]: Submitter approaching timeout -> restarting self
     171Submitted batch job 3303033
     172}}}
     173
     174==== Listing of the final checkpoint submitter log - completed workflow ====
     175
     176Here is the listing of the final log file, '''log_submitter_3303041.out''', generated by [wiki://Workshops/JobCheckpointing/Examples#checkpoint_submitter.sh checkpoint_submitter.sh] ending with the message "Workflow complete. Exiting submitter."
     177
     178{{{
     179[tulaneID@cypress2 ~]$cat log_submitter_3303041.out
     180Info[20260401-16:14:35]: Start on cypress01-066; JOBID=3303041
     181Info[20260401-16:14:35]: Submitter settings:
     182Info[20260401-16:14:35]: CKPT_PATH=state_iter_py.txt
     183Info[20260401-16:14:35]: MAX_ITER=500
     184Info[20260401-16:14:35]: RUNNER_SCRIPT=checkpoint_runner.sh
     185Info[20260401-16:14:35]: SUBMITTER_MARGIN_SEC=60
     186Info[20260401-16:14:35]: SUBMITTER_TIME_LIMIT=00:03:30
     187Info[20260401-16:14:35]: Loaded state: submitter_cycle=5
     188Info[20260401-16:14:35]: Submitter cycle: 5
     189Info[20260401-16:14:35]: Current checkpoint iteration: 501
     190Info[20260401-16:14:35]: Reached MAX_ITER=500; running final analytics.
     191Info[20260401-16:14:35]: Running analytics aggregate_usage.py ...
     192Wrote worker_usage.csv
     193     JobID  ElapsedSeconds   TotalCPU  ... ReqMem       NodeList      State
     1940  3303032             121  00:00.389  ...  512Mn  cypress01-066     FAILED
     1951  3303034             120  00:00.388  ...  512Mn  cypress01-066     FAILED
     1962  3303036             121  00:00.389  ...  512Mn  cypress01-066     FAILED
     1973  3303038             120  00:00.391  ...  512Mn  cypress01-066     FAILED
     1984  3303040              22  00:00.374  ...  512Mn  cypress01-066  COMPLETED
     199
     200[5 rows x 9 columns]
     201Info[20260401-16:14:36]: Running analytics plot_progress.py ...
     202Saved progress_with_anomalies.png
     203Saved resource_trend.png
     204Saved anomaly_details.csv
     205Saved anomaly_stats_table.png
     206Info[20260401-16:14:41]: Workflow complete. Exiting submitter.
     207}}}
     208
     209=== Sample checkpoint runner logs ===
     210
     211Here are the first and last output logs of the sequence of jobs generated by the job script [wiki://Workshops/JobCheckpointing/Examples#checkpoint_runner.sh checkpoint_runner.sh].
     212
     213All of the jobs in that sequence were submitted by the [wiki://Workshops/JobCheckpointing/Examples#checkpoint_submitter.sh checkpoint_submitter.sh] job script.
     214
     215==== Listing of the initial checkpoint runner log - starting application ====
     216
     217Here is the listing of the intial log file, '''log_runner_3303032.out''', generated by the job script [wiki://Workshops/JobCheckpointing/Examples#checkpoint_runner.sh checkpoint_runner.sh] ending with message "Timeout + checkpoint advance detected (0->120)."
     218
     219{{{
     220[tulaneID@cypress2 ~]$cat log_runner_3303032.out
     221Info[20260401-16:01:13]: Start on cypress01-066; JOBID=3303032
     222Info[20260401-16:01:13]: Runner settings:
     223Info[20260401-16:01:13]: APP_CMD=python3 checkpoint_signal_iter.py
     224Info[20260401-16:01:13]: CHECKPOINT_EVERY=20
     225Info[20260401-16:01:13]: CKPT_PATH=state_iter_py.txt
     226Info[20260401-16:01:13]: LAUNCH_MODE=direct
     227Info[20260401-16:01:13]: MAX_ITER=500
     228Info[20260401-16:01:13]: MODULE_LIST=anaconda3/2023.07
     229Info[20260401-16:01:13]: RUNNER_MARGIN_SEC=60
     230Info[20260401-16:01:13]: RUNNER_TIME_LIMIT=00:03:00
     231Info[20260401-16:01:13]: SRUN_ARGS=-n 1
     232Info[20260401-16:01:13]: Loading modules.
    129233Resuming from i=0 (iteration-based every 20)
    130234[periodic/iter] saved i=20
    … …  
    134238[periodic/iter] saved i=100
    135239SIGTERM: saving i=120 and exiting 99
    136 Info[20260313-22:20:17]: Program exit code (from timeout wrapper): 124
    137 Info[20260313-22:20:17]: Timeout TERM observed; checkpoint advanced (0->120). Requeueing...
    138 Info[20260313-22:20:17]: Requeued via scontrol.
    139 Info[20260313-22:20:57]: Start on cypress01-066; JOB_ID=3300699; RESTARTS=1
    140 Info[20260313-22:20:57]: Settings:
    141 Info[20260313-22:20:57]: MODULE_LIST=anaconda3/2023.07
    142 Info[20260313-22:20:57]: APP_CMD=python3 checkpoint_signal_iter.py
    143 Info[20260313-22:20:57]: LAUNCH_MODE=direct
    144 Info[20260313-22:20:57]: SRUN_ARGS=-n 1
    145 Info[20260313-22:20:57]: TIME_LIMIT=00:03:00
    146 Info[20260313-22:20:57]: MARGIN_SEC=60
    147 Info[20260313-22:20:57]: CKPT_PATH=state_iter_py.txt
    148 Info[20260313-22:20:57]: CHECKPOINT_EVERY=20
    149 Info[20260313-22:20:57]: MAX_ITER=500
    150 Info[20260313-22:20:57]: MAX_RESTARTS=10
    151 === BEGIN JOB SNAPSHOT (scontrol) ===
    152 JobId=3300699 Name=ckpt_requeue_demo
    153    Priority=80808 Nice=0 Account=<groupID> QOS=normal
    154    JobState=RUNNING Reason=None Dependency=(null)
    155    Requeue=1 Restarts=1 BatchFlag=1 ExitCode=0:0
    156    RunTime=00:00:00 TimeLimit=00:03:00 TimeMin=N/A
    157    StartTime=2026-03-13T22:20:57 EndTime=2026-03-13T22:23:57
    158    Partition=centos7 AllocNode:Sid=cypress2:33768
    159 === END JOB SNAPSHOT (scontrol) ===
    160 Resuming from i=120 (iteration-based every 20)
    161 [periodic/iter] saved i=140
    162 [periodic/iter] saved i=160
    163 [periodic/iter] saved i=180
    164 [periodic/iter] saved i=200
    165 [periodic/iter] saved i=220
    166 SIGTERM: saving i=240 and exiting 99
    167 Info[20260313-22:22:57]: Program exit code (from timeout wrapper): 124
    168 Info[20260313-22:22:57]: Timeout TERM observed; checkpoint advanced (120->240). Requeueing...
    169 Info[20260313-22:22:57]: Requeued via scontrol.
    170 Info[20260313-22:23:27]: Start on cypress01-066; JOB_ID=3300699; RESTARTS=2
    171 Info[20260313-22:23:27]: Settings:
    172 Info[20260313-22:23:27]: MODULE_LIST=anaconda3/2023.07
    173 Info[20260313-22:23:27]: APP_CMD=python3 checkpoint_signal_iter.py
    174 Info[20260313-22:23:27]: LAUNCH_MODE=direct
    175 Info[20260313-22:23:27]: SRUN_ARGS=-n 1
    176 Info[20260313-22:23:27]: TIME_LIMIT=00:03:00
    177 Info[20260313-22:23:27]: MARGIN_SEC=60
    178 Info[20260313-22:23:27]: CKPT_PATH=state_iter_py.txt
    179 Info[20260313-22:23:27]: CHECKPOINT_EVERY=20
    180 Info[20260313-22:23:27]: MAX_ITER=500
    181 Info[20260313-22:23:27]: MAX_RESTARTS=10
    182 === BEGIN JOB SNAPSHOT (scontrol) ===
    183 JobId=3300699 Name=ckpt_requeue_demo
    184    Priority=80808 Nice=0 Account=<groupID> QOS=normal
    185    JobState=RUNNING Reason=None Dependency=(null)
    186    Requeue=1 Restarts=2 BatchFlag=1 ExitCode=0:0
    187    RunTime=00:00:00 TimeLimit=00:03:00 TimeMin=N/A
    188    StartTime=2026-03-13T22:23:27 EndTime=2026-03-13T22:26:27
    189    Partition=centos7 AllocNode:Sid=cypress2:33768
    190 === END JOB SNAPSHOT (scontrol) ===
    191 Resuming from i=240 (iteration-based every 20)
    192 [periodic/iter] saved i=260
    193 [periodic/iter] saved i=280
    194 [periodic/iter] saved i=300
    195 [periodic/iter] saved i=320
    196 [periodic/iter] saved i=340
    197 SIGTERM: saving i=360 and exiting 99
    198 Info[20260313-22:25:27]: Program exit code (from timeout wrapper): 124
    199 Info[20260313-22:25:27]: Timeout TERM observed; checkpoint advanced (240->360). Requeueing...
    200 Info[20260313-22:25:27]: Requeued via scontrol.
    201 Info[20260313-22:25:57]: Start on cypress01-066; JOB_ID=3300699; RESTARTS=3
    202 Info[20260313-22:25:57]: Settings:
    203 Info[20260313-22:25:57]: MODULE_LIST=anaconda3/2023.07
    204 Info[20260313-22:25:57]: APP_CMD=python3 checkpoint_signal_iter.py
    205 Info[20260313-22:25:57]: LAUNCH_MODE=direct
    206 Info[20260313-22:25:57]: SRUN_ARGS=-n 1
    207 Info[20260313-22:25:57]: TIME_LIMIT=00:03:00
    208 Info[20260313-22:25:57]: MARGIN_SEC=60
    209 Info[20260313-22:25:57]: CKPT_PATH=state_iter_py.txt
    210 Info[20260313-22:25:57]: CHECKPOINT_EVERY=20
    211 Info[20260313-22:25:57]: MAX_ITER=500
    212 Info[20260313-22:25:57]: MAX_RESTARTS=10
    213 === BEGIN JOB SNAPSHOT (scontrol) ===
    214 JobId=3300699 Name=ckpt_requeue_demo
    215    Priority=80808 Nice=0 Account=<groupID> QOS=normal
    216    JobState=RUNNING Reason=None Dependency=(null)
    217    Requeue=1 Restarts=3 BatchFlag=1 ExitCode=0:0
    218    RunTime=00:00:00 TimeLimit=00:03:00 TimeMin=N/A
    219    StartTime=2026-03-13T22:25:57 EndTime=2026-03-13T22:28:57
    220    Partition=centos7 AllocNode:Sid=cypress2:33768
    221 === END JOB SNAPSHOT (scontrol) ===
    222 Resuming from i=360 (iteration-based every 20)
    223 [periodic/iter] saved i=380
    224 [periodic/iter] saved i=400
    225 [periodic/iter] saved i=420
    226 [periodic/iter] saved i=440
    227 [periodic/iter] saved i=460
    228 SIGTERM: saving i=480 and exiting 99
    229 Info[20260313-22:27:57]: Program exit code (from timeout wrapper): 124
    230 Info[20260313-22:27:57]: Timeout TERM observed; checkpoint advanced (360->480). Requeueing...
    231 Info[20260313-22:27:57]: Requeued via scontrol.
    232 Info[20260313-22:28:20]: Start on cypress01-066; JOB_ID=3300699; RESTARTS=4
    233 Info[20260313-22:28:20]: Settings:
    234 Info[20260313-22:28:20]: MODULE_LIST=anaconda3/2023.07
    235 Info[20260313-22:28:20]: APP_CMD=python3 checkpoint_signal_iter.py
    236 Info[20260313-22:28:20]: LAUNCH_MODE=direct
    237 Info[20260313-22:28:20]: SRUN_ARGS=-n 1
    238 Info[20260313-22:28:20]: TIME_LIMIT=00:03:00
    239 Info[20260313-22:28:20]: MARGIN_SEC=60
    240 Info[20260313-22:28:20]: CKPT_PATH=state_iter_py.txt
    241 Info[20260313-22:28:20]: CHECKPOINT_EVERY=20
    242 Info[20260313-22:28:20]: MAX_ITER=500
    243 Info[20260313-22:28:20]: MAX_RESTARTS=10
    244 === BEGIN JOB SNAPSHOT (scontrol) ===
    245 JobId=3300699 Name=ckpt_requeue_demo
    246    Priority=80808 Nice=0 Account=<groupID> QOS=normal
    247    JobState=RUNNING Reason=None Dependency=(null)
    248    Requeue=1 Restarts=4 BatchFlag=1 ExitCode=0:0
    249    RunTime=00:00:02 TimeLimit=00:03:00 TimeMin=N/A
    250    StartTime=2026-03-13T22:28:18 EndTime=2026-03-13T22:31:18
    251    Partition=centos7 AllocNode:Sid=cypress2:33768
    252 === END JOB SNAPSHOT (scontrol) ===
     240Info[20260401-16:03:13]: Program exit code (from timeout wrapper): 124
     241Info[20260401-16:03:13]: Timeout + checkpoint advance detected (0->120).
     242}}}
     243
     244==== Listing of the final checkpoint runner log - completed application ====
     245
     246Here is the listing of the final log file, '''log_runner_3303040.out''', generated by [wiki://Workshops/JobCheckpointing/Examples#checkpoint_runner.sh checkpoint_runner.sh] ending with message "Application finished all iterations."
     247
     248{{{
     249[tulaneID@cypress2 ~]$cat log_runner_3303040.out
     250Info[20260401-16:11:54]: Start on cypress01-066; JOBID=3303040
     251Info[20260401-16:11:54]: Runner settings:
     252Info[20260401-16:11:54]: APP_CMD=python3 checkpoint_signal_iter.py
     253Info[20260401-16:11:54]: CHECKPOINT_EVERY=20
     254Info[20260401-16:11:54]: CKPT_PATH=state_iter_py.txt
     255Info[20260401-16:11:54]: LAUNCH_MODE=direct
     256Info[20260401-16:11:54]: MAX_ITER=500
     257Info[20260401-16:11:54]: MODULE_LIST=anaconda3/2023.07
     258Info[20260401-16:11:54]: RUNNER_MARGIN_SEC=60
     259Info[20260401-16:11:54]: RUNNER_TIME_LIMIT=00:03:00
     260Info[20260401-16:11:55]: SRUN_ARGS=-n 1
     261Info[20260401-16:11:55]: Loading modules.
    253262Resuming from i=480 (iteration-based every 20)
    254263[periodic/iter] saved i=500
    255264Reached i=501 > 500; exiting 0
    256 Info[20260313-22:28:41]: Program exit code (from timeout wrapper): 0
    257 Info[20260313-22:28:41]: Completed.
    258 }}}
     265Info[20260401-16:12:16]: Program exit code (from timeout wrapper): 0
     266Info[20260401-16:12:16]: Application finished all iterations.
     267}}}
     268
     269=== Listings of post-workflow analysis output files ===
     270
     271The following is the complete, annotated listing of all the generated .csv and .txt files
     272
     273{{{
     274[tulaneID@cypress2 ~]$tail -n +1 *.csv *.txt
     275==> anomaly_details.csv <==
     276JobID,StartTime,ElapsedSeconds,MaxRSS_MiB,TimeZ,MemModZ,AnomalyType
     2773303032,2026-04-01 16:01:13,121.0,14.88494873046875,0.5126573320092711,0.0,none
     2783303034,2026-04-01 16:03:53,120.0,14.8773193359375,0.487278256167228,0.0,none
     2793303036,2026-04-01 16:06:33,121.0,14.8773193359375,0.5126573320092711,0.0,none
     2803303038,2026-04-01 16:09:14,120.0,14.881134033203123,0.487278256167228,0.0,none
     2813303040,2026-04-01 16:11:54,22.0,13.06915283203125,-1.999871176352998,639.4260000005955,memory
     282
     283==> worker_usage.csv <==
     284JobID,ElapsedSeconds,TotalCPU,MaxRSS,MaxRSS_bytes,MaxRSS_MiB,ReqMem,NodeList,State
     2853303032,121,00:00.389,15608K,15608000.0,14.88494873046875,512Mn,cypress01-066,FAILED
     2863303034,120,00:00.388,15600K,15600000.0,14.8773193359375,512Mn,cypress01-066,FAILED
     2873303036,121,00:00.389,15600K,15600000.0,14.8773193359375,512Mn,cypress01-066,FAILED
     2883303038,120,00:00.391,15604K,15604000.0,14.881134033203125,512Mn,cypress01-066,FAILED
     2893303040,22,00:00.374,13704K,13704000.0,13.06915283203125,512Mn,cypress01-066,COMPLETED
     290
     291==> state_iter_py.txt <==
     292501
     293==> submitter_state.txt <==
     294submitter_cycle=5
     295
     296==> worker_ids.txt <==
     2973303032
     2983303034
     2993303036
     3003303038
     3013303040
     302}}}
     303
     304=== Post-workflow analysis plots ===
     305
     306==== Background ====
     307
     308For background on the following plots, see [wiki://Workshops/JobCheckpointing/Examples#Plottingexecutiontimeandmemoryusage Plotting execution time and memory usage].
     309
     310==== Table of anomalies, compute time and memory usage ====
     311
     312Here is the plot, '''anomaly_stats_table.png''', showing the table of anomalies, if any, from among all of the [wiki://Workshops/JobCheckpointing/Examples#checkpoint_runner.sh checkpoint_runner.sh] jobs.
     313
     314[[Image(python_anomaly_stats_table.png)]]
     315
     316==== Compute progress ====
     317
     318Here is the plot, '''progress_with_anomalies.png''', showing '''Checkpoint Progress''' and '''MaxRSS''' (SLURM max job memory usage) vs '''Wall Time'''. Anomalies, if any, would be overplotted in red.
     319
     320[[Image(python_progress_with_anomalies.png, width=800px)]]
     321
     322==== Execution time and memory usage trends ====
     323
     324Here is the plot, '''resource_trend.png''', showing '''Elapsed (time)''' and '''MaxRSS''' vs. '''Worker job index'''.
     325
     326[[Image(python_resource_trend.png, width=800px)]]