Changes between Version 3 and Version 4 of Workshops/JobCheckpointing/Examples/BASH


Ignore:
Timestamp:
04/02/2026 07:26:50 PM (4 months ago)
Author:
Carl Baribault
Comment:

Updated output for submitter/runner

Legend:

Unmodified
Added
Removed
Modified

  • Workshops/JobCheckpointing/Examples/BASH

    v3 v4  
    22= BASH Checkpointing Example =
    33
    4 == Checkpoint Runner ==
    5 
    6 See [wiki:Workshops/JobCheckpointing/Examples#CheckpointRunner Checkpoint Runner] for the contents of the job script file '''checkpoint_runner.sh'''.
    7 
    84== BASH Checkpointing Application ==
     5
     6Here is an example of a checkpointing application implemented in BASH script.
     7
     8To run the example, see [wiki://Workshops/JobCheckpointing/Examples/BASH#RunningtheBASHcheckpointingexampleonCypress Running the BASH checkpointing example on Cypress] below.
    99
    1010=== checkpoint_signal_iter.sh ===
    … …  
    6565}}}
    6666
    67 == Running BASH checkpointing example on Cypress ==
     67== Running the BASH checkpointing example on Cypress ==
    6868
    6969To run the BASH checkpointing job example, defaulting to checkpointing every 20 application iterations and a total of 500 iterations, perform the following.
    7070
    71 1. Edit the files '''checkpoint_runner.sh''' and '''checkpoint_signal_iter.sh''' as shown above in your current directory.
    72  For file editing with nano, etc., see [[https://wiki.hpc.tulane.edu/trac/wiki/cypress/FileEditingSoftware/Example|File Editing Example]].
     711. Create your own versions of the following files in your current directory on Cypress. (For file editing with '''nano''' on Cypress, see [[https://wiki.hpc.tulane.edu/trac/wiki/cypress/FileEditingSoftware/Example|File Editing Example]].)
     72
     73 * [wiki://Workshops/JobCheckpointing/Examples#checkpoint_submitter.sh checkpoint_submitter.sh]
     74 * [wiki://Workshops/JobCheckpointing/Examples#checkpoint_runner.sh checkpoint_runner.sh]
     75 * [wiki://Workshops/JobCheckpointing/Examples/BASH#checkpoint_signal_iter.sh checkpoint_signal_iter.sh]
    7376
    74772. Change permissions on the BASH application script, '''checkpoint_signal_iter.sh''' executable via the following command.
    … …  
    8184
    8285{{{
    83 [tulaneID@cypress1 ~]$ APP_CMD="./checkpoint_signal_iter.sh" CKPT_PATH=state_iter_bash.txt sbatch checkpoint_runner.sh
    84 }}}
    85 
    86 4. Monitor the job's output via the following command, substituting the job ID for <jobID>.
    87 
    88 {{{
    89 [tulaneID@cypress1 ~]$ tail -f log_<jobID>.*
    90 }}}
    91 
    92 5. Here are normal results for the output and error files, '''log_<jobID>.err''' and  '''log_<jobID>.out''', observing that the job cancelled and requeued itself many times.
    93 
    94 {{{
    95 [tulaneID@cypress1 ~]$cat log_3300698.err
    96 Terminated
    97 slurmstepd: *** JOB 3300698 CANCELLED AT 2026-03-13T22:20:11 ***
    98 Terminated
    99 slurmstepd: *** JOB 3300698 CANCELLED AT 2026-03-13T22:22:27 ***
    100 Terminated
    101 slurmstepd: *** JOB 3300698 CANCELLED AT 2026-03-13T22:24:57 ***
    102 Terminated
    103 slurmstepd: *** JOB 3300698 CANCELLED AT 2026-03-13T22:27:27 ***
    104 }}}
    105  
    106 {{{
    107 [tulaneID@cypress1 ~]$cat log_3300698.out
    108 Info[20260313-22:18:11]: Start on cypress01-066; JOB_ID=3300698; RESTARTS=0
    109 Info[20260313-22:18:11]: Settings:
    110 Info[20260313-22:18:11]: MODULE_LIST=anaconda3/2023.07
    111 Info[20260313-22:18:11]: APP_CMD=./checkpoint_signal_iter.sh
    112 Info[20260313-22:18:11]: LAUNCH_MODE=direct
    113 Info[20260313-22:18:11]: SRUN_ARGS=-n 1
    114 Info[20260313-22:18:11]: TIME_LIMIT=00:03:00
    115 Info[20260313-22:18:11]: MARGIN_SEC=60
    116 Info[20260313-22:18:11]: CKPT_PATH=state_iter_bash.txt
    117 Info[20260313-22:18:11]: CHECKPOINT_EVERY=20
    118 Info[20260313-22:18:11]: MAX_ITER=500
    119 Info[20260313-22:18:11]: MAX_RESTARTS=10
    120 === BEGIN JOB SNAPSHOT (scontrol) ===
    121 JobId=3300698 Name=ckpt_requeue_demo
    122    Priority=80808 Nice=0 Account=<groupID> QOS=normal
    123    JobState=RUNNING Reason=None Dependency=(null)
    124    Requeue=1 Restarts=0 BatchFlag=1 ExitCode=0:0
    125    RunTime=00:00:01 TimeLimit=00:03:00 TimeMin=N/A
    126    StartTime=2026-03-13T22:18:10 EndTime=2026-03-13T22:21:10
    127    Partition=centos7 AllocNode:Sid=cypress2:33768
    128 === END JOB SNAPSHOT (scontrol) ===
    129 Resuming from i=0 (every 20, MAX_ITER=500)
    130 [periodic/iter] saved i=20
    131 [periodic/iter] saved i=40
    132 [periodic/iter] saved i=60
    133 [periodic/iter] saved i=80
    134 [periodic/iter] saved i=100
    135 Bash SIGTERM: saving i=120 and exiting 99
    136 Info[20260313-22:20:11]: Program exit code (from timeout wrapper): 124
    137 Info[20260313-22:20:11]: Timeout TERM observed; checkpoint advanced (0->120). Requeueing...
    138 Info[20260313-22:20:11]: Requeued via scontrol.
    139 Info[20260313-22:20:27]: Start on cypress01-066; JOB_ID=3300698; RESTARTS=1
    140 Info[20260313-22:20:27]: Settings:
    141 Info[20260313-22:20:27]: MODULE_LIST=anaconda3/2023.07
    142 Info[20260313-22:20:27]: APP_CMD=./checkpoint_signal_iter.sh
    143 Info[20260313-22:20:27]: LAUNCH_MODE=direct
    144 Info[20260313-22:20:27]: SRUN_ARGS=-n 1
    145 Info[20260313-22:20:27]: TIME_LIMIT=00:03:00
    146 Info[20260313-22:20:27]: MARGIN_SEC=60
    147 Info[20260313-22:20:27]: CKPT_PATH=state_iter_bash.txt
    148 Info[20260313-22:20:27]: CHECKPOINT_EVERY=20
    149 Info[20260313-22:20:27]: MAX_ITER=500
    150 Info[20260313-22:20:27]: MAX_RESTARTS=10
    151 === BEGIN JOB SNAPSHOT (scontrol) ===
    152 JobId=3300698 Name=ckpt_requeue_demo
    153    Priority=80808 Nice=0 Account=<groupID> QOS=normal
    154    JobState=RUNNING Reason=None Dependency=(null)
    155    Requeue=1 Restarts=1 BatchFlag=1 ExitCode=0:0
    156    RunTime=00:00:00 TimeLimit=00:03:00 TimeMin=N/A
    157    StartTime=2026-03-13T22:20:27 EndTime=2026-03-13T22:23:27
    158    Partition=centos7 AllocNode:Sid=cypress2:33768
    159 === END JOB SNAPSHOT (scontrol) ===
    160 Resuming from i=120 (every 20, MAX_ITER=500)
    161 [periodic/iter] saved i=140
    162 [periodic/iter] saved i=160
    163 [periodic/iter] saved i=180
    164 [periodic/iter] saved i=200
    165 [periodic/iter] saved i=220
    166 Bash SIGTERM: saving i=240 and exiting 99
    167 Info[20260313-22:22:27]: Program exit code (from timeout wrapper): 124
    168 Info[20260313-22:22:27]: Timeout TERM observed; checkpoint advanced (120->240). Requeueing...
    169 Info[20260313-22:22:27]: Requeued via scontrol.
    170 Info[20260313-22:22:57]: Start on cypress01-066; JOB_ID=3300698; RESTARTS=2
    171 Info[20260313-22:22:57]: Settings:
    172 Info[20260313-22:22:57]: MODULE_LIST=anaconda3/2023.07
    173 Info[20260313-22:22:57]: APP_CMD=./checkpoint_signal_iter.sh
    174 Info[20260313-22:22:57]: LAUNCH_MODE=direct
    175 Info[20260313-22:22:57]: SRUN_ARGS=-n 1
    176 Info[20260313-22:22:57]: TIME_LIMIT=00:03:00
    177 Info[20260313-22:22:57]: MARGIN_SEC=60
    178 Info[20260313-22:22:57]: CKPT_PATH=state_iter_bash.txt
    179 Info[20260313-22:22:57]: CHECKPOINT_EVERY=20
    180 Info[20260313-22:22:57]: MAX_ITER=500
    181 Info[20260313-22:22:57]: MAX_RESTARTS=10
    182 === BEGIN JOB SNAPSHOT (scontrol) ===
    183 JobId=3300698 Name=ckpt_requeue_demo
    184    Priority=80808 Nice=0 Account=<groupID> QOS=normal
    185    JobState=RUNNING Reason=None Dependency=(null)
    186    Requeue=1 Restarts=2 BatchFlag=1 ExitCode=0:0
    187    RunTime=00:00:00 TimeLimit=00:03:00 TimeMin=N/A
    188    StartTime=2026-03-13T22:22:57 EndTime=2026-03-13T22:25:57
    189    Partition=centos7 AllocNode:Sid=cypress2:33768
    190 === END JOB SNAPSHOT (scontrol) ===
    191 Resuming from i=240 (every 20, MAX_ITER=500)
    192 [periodic/iter] saved i=260
    193 [periodic/iter] saved i=280
    194 [periodic/iter] saved i=300
    195 [periodic/iter] saved i=320
    196 [periodic/iter] saved i=340
    197 Bash SIGTERM: saving i=360 and exiting 99
    198 Info[20260313-22:24:57]: Program exit code (from timeout wrapper): 124
    199 Info[20260313-22:24:57]: Timeout TERM observed; checkpoint advanced (240->360). Requeueing...
    200 Info[20260313-22:24:57]: Requeued via scontrol.
    201 Info[20260313-22:25:27]: Start on cypress01-066; JOB_ID=3300698; RESTARTS=3
    202 Info[20260313-22:25:27]: Settings:
    203 Info[20260313-22:25:27]: MODULE_LIST=anaconda3/2023.07
    204 Info[20260313-22:25:27]: APP_CMD=./checkpoint_signal_iter.sh
    205 Info[20260313-22:25:27]: LAUNCH_MODE=direct
    206 Info[20260313-22:25:27]: SRUN_ARGS=-n 1
    207 Info[20260313-22:25:27]: TIME_LIMIT=00:03:00
    208 Info[20260313-22:25:27]: MARGIN_SEC=60
    209 Info[20260313-22:25:27]: CKPT_PATH=state_iter_bash.txt
    210 Info[20260313-22:25:27]: CHECKPOINT_EVERY=20
    211 Info[20260313-22:25:27]: MAX_ITER=500
    212 Info[20260313-22:25:27]: MAX_RESTARTS=10
    213 === BEGIN JOB SNAPSHOT (scontrol) ===
    214 JobId=3300698 Name=ckpt_requeue_demo
    215    Priority=80808 Nice=0 Account=<groupID> QOS=normal
    216    JobState=RUNNING Reason=None Dependency=(null)
    217    Requeue=1 Restarts=3 BatchFlag=1 ExitCode=0:0
    218    RunTime=00:00:00 TimeLimit=00:03:00 TimeMin=N/A
    219    StartTime=2026-03-13T22:25:27 EndTime=2026-03-13T22:28:27
    220    Partition=centos7 AllocNode:Sid=cypress2:33768
    221 === END JOB SNAPSHOT (scontrol) ===
    222 Resuming from i=360 (every 20, MAX_ITER=500)
    223 [periodic/iter] saved i=380
    224 [periodic/iter] saved i=400
    225 [periodic/iter] saved i=420
    226 [periodic/iter] saved i=440
    227 [periodic/iter] saved i=460
    228 Bash SIGTERM: saving i=480 and exiting 99
    229 Info[20260313-22:27:27]: Program exit code (from timeout wrapper): 124
    230 Info[20260313-22:27:27]: Timeout TERM observed; checkpoint advanced (360->480). Requeueing...
    231 Info[20260313-22:27:27]: Requeued via scontrol.
    232 Info[20260313-22:27:57]: Start on cypress01-066; JOB_ID=3300698; RESTARTS=4
    233 Info[20260313-22:27:57]: Settings:
    234 Info[20260313-22:27:57]: MODULE_LIST=anaconda3/2023.07
    235 Info[20260313-22:27:57]: APP_CMD=./checkpoint_signal_iter.sh
    236 Info[20260313-22:27:57]: LAUNCH_MODE=direct
    237 Info[20260313-22:27:57]: SRUN_ARGS=-n 1
    238 Info[20260313-22:27:57]: TIME_LIMIT=00:03:00
    239 Info[20260313-22:27:57]: MARGIN_SEC=60
    240 Info[20260313-22:27:57]: CKPT_PATH=state_iter_bash.txt
    241 Info[20260313-22:27:57]: CHECKPOINT_EVERY=20
    242 Info[20260313-22:27:57]: MAX_ITER=500
    243 Info[20260313-22:27:57]: MAX_RESTARTS=10
    244 === BEGIN JOB SNAPSHOT (scontrol) ===
    245 JobId=3300698 Name=ckpt_requeue_demo
    246    Priority=80808 Nice=0 Account=<groupID> QOS=normal
    247    JobState=RUNNING Reason=None Dependency=(null)
    248    Requeue=1 Restarts=4 BatchFlag=1 ExitCode=0:0
    249    RunTime=00:00:00 TimeLimit=00:03:00 TimeMin=N/A
    250    StartTime=2026-03-13T22:27:57 EndTime=2026-03-13T22:30:57
    251    Partition=centos7 AllocNode:Sid=cypress2:33768
    252 === END JOB SNAPSHOT (scontrol) ===
     86[tulaneID@cypress1 ~]$ APP_CMD="./checkpoint_signal_iter.sh" \
     87CKPT_PATH=state_iter_bash.txt \
     88sbatch checkpoint_runner.sh
     89}}}
     90
     914. You can monitor the jobs' outputs as they are updated by repeatedly executing the following command, using Ctrl+C to exit files as they finish updating.
     92
     93{{{
     94[tulaneID@cypress1 ~]$ tail -f log_*
     95}}}
     96
     97== Sample run outputs ==
     98
     99=== List of output files ===
     100
     101Here is a list of all output files from a sample run sorted by time, latest-to-earliest
     102
     103{{{
     104[tulaneID@cypress2 ~]$ls -lt *.csv *.err *.out *.png *.txt
     105-rw-r--r-- 1 tulaneID groupID   452 Apr  1 17:57 anomaly_details.csv
     106-rw-r--r-- 1 tulaneID groupID 26376 Apr  1 17:57 anomaly_stats_table.png
     107-rw-r--r-- 1 tulaneID groupID  1402 Apr  1 17:57 log_submitter_3303154.out
     108-rw-r--r-- 1 tulaneID groupID 40571 Apr  1 17:57 resource_trend.png
     109-rw-r--r-- 1 tulaneID groupID 52107 Apr  1 17:57 progress_with_anomalies.png
     110-rw-r--r-- 1 tulaneID groupID   495 Apr  1 17:57 worker_usage.csv
     111-rw-r--r-- 1 tulaneID groupID   461 Apr  1 17:57 log_submitter_3303150.err
     112-rw-r--r-- 1 tulaneID groupID   761 Apr  1 17:57 log_submitter_3303150.out
     113-rw-r--r-- 1 tulaneID groupID     0 Apr  1 17:57 log_submitter_3303154.err
     114-rw-r--r-- 1 tulaneID groupID    18 Apr  1 17:57 submitter_state.txt
     115-rw-r--r-- 1 tulaneID groupID   818 Apr  1 17:54 log_runner_3303151.out
     116-rw------- 1 tulaneID groupID     4 Apr  1 17:54 state_iter_bash.txt
     117-rw-r--r-- 1 tulaneID groupID     0 Apr  1 17:54 log_runner_3303151.err
     118-rw-r--r-- 1 tulaneID groupID   491 Apr  1 17:54 log_submitter_3303148.err
     119-rw-r--r-- 1 tulaneID groupID   761 Apr  1 17:54 log_submitter_3303148.out
     120-rw-r--r-- 1 tulaneID groupID    40 Apr  1 17:54 worker_ids.txt
     121-rw-r--r-- 1 tulaneID groupID    11 Apr  1 17:53 log_runner_3303149.err
     122-rw-r--r-- 1 tulaneID groupID   956 Apr  1 17:53 log_runner_3303149.out
     123-rw-r--r-- 1 tulaneID groupID   461 Apr  1 17:51 log_submitter_3303146.err
     124-rw-r--r-- 1 tulaneID groupID   761 Apr  1 17:51 log_submitter_3303146.out
     125-rw-r--r-- 1 tulaneID groupID    11 Apr  1 17:51 log_runner_3303147.err
     126-rw-r--r-- 1 tulaneID groupID   956 Apr  1 17:51 log_runner_3303147.out
     127-rw-r--r-- 1 tulaneID groupID   491 Apr  1 17:49 log_submitter_3303142.err
     128-rw-r--r-- 1 tulaneID groupID   761 Apr  1 17:49 log_submitter_3303142.out
     129-rw-r--r-- 1 tulaneID groupID    11 Apr  1 17:48 log_runner_3303143.err
     130-rw-r--r-- 1 tulaneID groupID   956 Apr  1 17:48 log_runner_3303143.out
     131-rw-r--r-- 1 tulaneID groupID   768 Apr  1 17:46 log_submitter_3303140.out
     132-rw-r--r-- 1 tulaneID groupID   491 Apr  1 17:46 log_submitter_3303140.err
     133-rw-r--r-- 1 tulaneID groupID    11 Apr  1 17:46 log_runner_3303141.err
     134-rw-r--r-- 1 tulaneID groupID   948 Apr  1 17:46 log_runner_3303141.out
     135}}}
     136
     137=== Listing of the final checkpoint submitter log - completed workflow ===
     138
     139The following is the listing of the final log file, '''log_submitter_3303154.out''', generated by '''checkpoint_submitter.sh''' ending with "Workflow complete. Exiting submitter."
     140
     141{{{
     142[tulaneID@cypress2 ~]$cat log_submitter_3303154.out
     143Info[20260401-17:57:08]: Start on cypress01-066; JOBID=3303154
     144Info[20260401-17:57:08]: Submitter settings:
     145Info[20260401-17:57:08]: CKPT_PATH=state_iter_bash.txt
     146Info[20260401-17:57:08]: MAX_ITER=500
     147Info[20260401-17:57:08]: RUNNER_SCRIPT=checkpoint_runner.sh
     148Info[20260401-17:57:08]: SUBMITTER_MARGIN_SEC=60
     149Info[20260401-17:57:08]: SUBMITTER_TIME_LIMIT=3:30
     150Info[20260401-17:57:08]: Loaded state: submitter_cycle=5
     151Info[20260401-17:57:08]: Submitter cycle: 5
     152Info[20260401-17:57:08]: Current checkpoint iteration: 501
     153Info[20260401-17:57:08]: Reached MAX_ITER=500; running final analytics.
     154Info[20260401-17:57:08]: Running analytics aggregate_usage.py ...
     155Wrote worker_usage.csv
     156     JobID  ElapsedSeconds   TotalCPU  ... ReqMem       NodeList      State
     1570  3303141             120  00:00.552  ...  512Mn  cypress01-066     FAILED
     1581  3303143             120  00:00.563  ...  512Mn  cypress01-066     FAILED
     1592  3303147             120  00:00.562  ...  512Mn  cypress01-066     FAILED
     1603  3303149             120  00:00.563  ...  512Mn  cypress01-066     FAILED
     1614  3303151              22  00:00.347  ...  512Mn  cypress01-066  COMPLETED
     162
     163[5 rows x 9 columns]
     164Info[20260401-17:57:09]: Running analytics plot_progress.py ...
     165Saved progress_with_anomalies.png
     166Saved resource_trend.png
     167Saved anomaly_details.csv
     168Saved anomaly_stats_table.png
     169Info[20260401-17:57:13]: Workflow complete. Exiting submitter.
     170}}}
     171
     172
     173=== Listing of the final checkpoint runner log - completed application ===
     174
     175The following is the listing of the final log file, '''log_runner_3303151.out''', generated by '''checkpoint_runner.sh''' ending with message "Application finished all iterations."
     176
     177{{{
     178[tulaneID@cypress2 ~]$cat log_runner_3303151.out
     179Info[20260401-17:54:37]: Start on cypress01-066; JOBID=3303151
     180Info[20260401-17:54:37]: Runner settings:
     181Info[20260401-17:54:37]: APP_CMD=./checkpoint_signal_iter.sh
     182Info[20260401-17:54:37]: CHECKPOINT_EVERY=20
     183Info[20260401-17:54:37]: CKPT_PATH=state_iter_bash.txt
     184Info[20260401-17:54:37]: LAUNCH_MODE=direct
     185Info[20260401-17:54:37]: MAX_ITER=500
     186Info[20260401-17:54:37]: MODULE_LIST=anaconda3/2023.07
     187Info[20260401-17:54:37]: RUNNER_MARGIN_SEC=60
     188Info[20260401-17:54:37]: RUNNER_TIME_LIMIT=00:03:00
     189Info[20260401-17:54:37]: SRUN_ARGS=-n 1
     190Info[20260401-17:54:37]: Loading modules.
    253191Resuming from i=480 (every 20, MAX_ITER=500)
    254192[periodic/iter] saved i=500
    255193Reached i=501 > 500; exiting 0
    256 Info[20260313-22:28:18]: Program exit code (from timeout wrapper): 0
    257 Info[20260313-22:28:18]: Completed.
    258 }}}
     194Info[20260401-17:54:59]: Program exit code (from timeout wrapper): 0
     195Info[20260401-17:54:59]: Application finished all iterations.
     196}}}
     197
     198
     199=== Listings of post-workflow analysis output files ===
     200
     201The following is the complete, annotated listing of all the generated .csv and .txt files
     202
     203{{{
     204[tulaneID@cypress2 ~]$tail -n +1 *.csv *.txt
     205==> anomaly_details.csv <==
     206JobID,StartTime,ElapsedSeconds,MaxRSS_MiB,TimeZ,MemModZ,AnomalyType
     2073303141,2026-04-01 17:44:05,120.0,3.726959228515625,0.49999999999999983,0.0,none
     2083303143,2026-04-01 17:46:46,120.0,3.73077392578125,0.49999999999999983,0.0,none
     2093303147,2026-04-01 17:49:26,120.0,3.73077392578125,0.49999999999999983,0.0,none
     2103303149,2026-04-01 17:51:57,120.0,3.73077392578125,0.49999999999999983,0.0,none
     2113303151,2026-04-01 17:54:37,22.0,3.7078857421875,-2.0,0.0,none
     212
     213==> worker_usage.csv <==
     214JobID,ElapsedSeconds,TotalCPU,MaxRSS,MaxRSS_bytes,MaxRSS_MiB,ReqMem,NodeList,State
     2153303141,120,00:00.552,3908K,3908000.0,3.726959228515625,512Mn,cypress01-066,FAILED
     2163303143,120,00:00.563,3912K,3912000.0,3.73077392578125,512Mn,cypress01-066,FAILED
     2173303147,120,00:00.562,3912K,3912000.0,3.73077392578125,512Mn,cypress01-066,FAILED
     2183303149,120,00:00.563,3912K,3912000.0,3.73077392578125,512Mn,cypress01-066,FAILED
     2193303151,22,00:00.347,3888K,3888000.0,3.7078857421875,512Mn,cypress01-066,COMPLETED
     220
     221==> state_iter_bash.txt <==
     222501
     223
     224==> submitter_state.txt <==
     225submitter_cycle=5
     226
     227==> worker_ids.txt <==
     2283303141
     2293303143
     2303303147
     2313303149
     2323303151
     233}}}
     234
     235=== Post-workflow analysis plots ===
     236
     237==== Anomaly stats, compute time and memory usage ====
     238
     239Here is the plot, anomaoly_stats_table.png, showing the table of anomalies in the use of compute time and memory.
     240
     241==== Compute progress ====
     242
     243Here is the plot, anomaoly_stats_table.png, showing '''Checkpoint Progress''' and SLURM memory usage, '''MaxRSS'''.
     244
     245==== Resource trend ====
     246
     247Here is the plot, resource_trend.png, showing '''Elapsed (time)''' and SLURM memory usage, '''MaxRSS'''.