[[PageOutline]]
= After your job has completed - determining cumulative resource efficiency =

== Assumptions ==
See [wiki:Workshops/JobParallelism/WhileYourJobIsRunning#Assumptions Assumptions] - same as for running jobs.

== Preliminary: tools available ==

=== On LONI clusters ===

LONI clusters provide the self-contained commands '''seff''' and '''qshow'''.

  * '''seff''' (See [https://github.com/SchedMD/slurm/tree/master/contribs/seff seff on github].)

  On LONI QB4 cluster:

 {{{
[loniID@qbd2 ~]$ seff -h
Usage: seff [Options] <Jobid>
       Options:
       -h    Help menu
       -v    Version
       -d    Debug mode: display raw Slurm data
[loniID@qbd2 ~]$ seff -v
seff Version 2.1
 }}}

   * '''qshow''' (provided by LONI)

  On LONI QB4 cluster:

 {{{
[loniID@qbd2 ~]$ qshow -h
** usage: qshow -n <options> <base-name> <begin #> <end #> <command>
...
Show and optionally kill user processes on remote nodes or execute
commands...
[loniID@qbd2 ~]$ qshow -v
qshow 2.74
 }}}

=== On Cypress ===

==== Local version of seff for SLURM 14 ====

We've provided a local version of the '''seff''' command in order to compensate for the earlier version of SLURM (v14.03.0) on Cypress where we've encountered insufficient support for the seff command as implemented on LONI clusters.

More specifically, this version of SLURM makes certain job performance values available only after the given job as completed.

==== The sacct command ====

The local version of seff available on Cypress uses the SLURM command '''sacct''' with more information available from the man page. (See '''man sacct'''.)

==== Result ====

Here is the result for seff on the previously running job which has now completed. 

{{{
[tulaneID@cypress1 ~]$seff 3336943
Job ID: 3336943
Cluster: cypress
User/Group: cbaribault/hpcstaff
State: COMPLETED (exit code 0:0)
Cores: 16
CPU Utilized: 0:00:20
CPU Efficiency: 1.8% of 00:01:11 core-walltime
Memory Utilized: 0.08 GB
Memory Efficiency: 0.2% of requested per-CPU memory x 16
------------------------------------------------------------
}}}

==== Summary - in contrast with the result for the running job ====

Admittedly, the above result for the completed job is stark contrast with the favorable result we obtained using our '''seffRunningPrototype''' while the job was still running.

Even though we've made an attempt at optimization in the R script '''bootstrapFutureApply.R''' (using R package '''future.apply''') compared to the R script '''bootstrap.R''' (using '''doParallel'''), the result above shows that we still have room for improvement of the overall efficiency of the job.

==== Fewer requested resources = faster job starts

In general, jobs that request fewer resources (memory, number of cores, number of nodes, or run time) will start more quickly others.

In other words, the SLURM resource manager will more frequently find an earlier - if not immediate - time slot to start the job as compared to other jobs requesting more resources.