After your job has completed - determining cumulative resource efficiency

Assumptions

See Assumptions - same as for running jobs.

Preliminary: tools available

On LONI clusters

LONI clusters provide the self-contained commands seff and qshow.

• seff (See ​seff on github.)

On LONI QB4 cluster:

[loniID@qbd2 ~]$ seff -h
Usage: seff [Options] <Jobid>
       Options:
       -h    Help menu
       -v    Version
       -d    Debug mode: display raw Slurm data
[loniID@qbd2 ~]$ seff -v
seff Version 2.1

• qshow (provided by LONI)

On LONI QB4 cluster:

[loniID@qbd2 ~]$ qshow -h
** usage: qshow -n <options> <base-name> <begin #> <end #> <command>
...
Show and optionally kill user processes on remote nodes or execute
commands...
[loniID@qbd2 ~]$ qshow -v
qshow 2.74

On Cypress

Local version of seff for SLURM 14

We've provided a local version of the seff command in order to compensate for the earlier version of SLURM (v14.03.0) on Cypress where we've encountered insufficient support for the seff command as implemented on LONI clusters.

More specifically, this version of SLURM makes certain job performance values available only after the given job as completed.

The sacct command

The local version of seff available on Cypress uses the SLURM command sacct with more information available from the man page. (See man sacct.)

Result

Here is the result for seff on the previously running job which has now completed.

[tulaneID@cypress1 ~]$seff 3336943
Job ID: 3336943
Cluster: cypress
User/Group: cbaribault/hpcstaff
State: COMPLETED (exit code 0:0)
Cores: 16
CPU Utilized: 0:00:20
CPU Efficiency: 1.8% of 00:01:11 core-walltime
Memory Utilized: 0.08 GB
Memory Efficiency: 0.2% of requested per-CPU memory x 16
------------------------------------------------------------

Summary - in contrast with the result for the running job

Admittedly, the above result for the completed job is stark contrast with the favorable result we obtained using our seffRunningPrototype while the job was still running.

Even though we've made an attempt at optimization in the R script bootstrapFutureApply.R (using R package future.apply) compared to the R script bootstrap.R (using doParallel), the result above shows that we still have room for improvement of the overall efficiency of the job.

Fewer requested resources = faster job starts

In general, jobs that request fewer resources (memory, number of cores, number of nodes, or run time) will start more quickly others.

In other words, the SLURM resource manager will more frequently find an earlier - if not immediate - time slot to start the job as compared to other jobs requesting more resources.