[[PageOutline]]
= While your job is running - determining current core efficiency =

== Assumptions ==
* Request sufficient processor resources

 For running jobs let's assume that you've requested sufficient processor resources via the following. (See '''man sbatch'''.)

[[br]]

 ||='''Description'''=||='''SBATCH options'''=||='''Default value'''=||='''Maximum value'''=||
 || # of nodes || -N, --nodes || Subject to -n and -c options || See [wiki:cypress/about#SLURMresourcemanager SLURM (resource manager)]||
 || # of tasks || -n, --ntasks || 1 task per node || 20 * (# of nodes) ||
 || # of cores/CPUs/processors per tasks || -c, --cpus-per-task || 1 core per task || 20 ||
 || total Random Access Memory (RAM) || --mem || 3200MB per core || 64/128/256GB ||
 || RAM per core || --mem-per-cpu || 3200MB || " " " ||

* Your job's memory requirement may be greater than your (# cores) * (Total RAM)/20.
 * For example, your job may require only 10 cores but all of the RAM available on a node with 128GB of RAM.

{{{
  ...
  #SBATCH --ntasks=1
  #SBATCH --cpus-per-task=10  # Use only 10 cores
  #SBATCH --mem=128           # Use all memory on 128GB node
  ...

}}}

== Current core efficiency for running jobs: (actual core usage) / (requested core allocation)

=== Example 1: an idev job for an idle interactive session ===

 1. '''Start an idev interactive session.'''

{{{
[tulaneID@cypress1 ~]$idev --partition=centos7
Requesting 1 node(s)  task(s) to normal queue of centos7 partition
1 task(s)/node, 20 cpu(s)/task, 0 MIC device(s)/node
Time: 0 (hr) 60 (min).
0d 0h 60m
Submitted batch job 3336903
JOBID=3336903 begin on cypress01-121
--> Creating interactive terminal session (login) on node cypress01-121.
--> You have 0 (hr) 60 (min).
--> Assigned Host List : /tmp/idev_nodes_file_cbaribault
Last login: Fri Dec  5 14:02:37 2025 from cypress2.cm.cluster
}}}

 '''For workshop''' using only 2 requested cores

{{{
[tulaneID@cypress1 ~]$idev --partition=workshop7 -c 2
}}}

 2. '''Login to Cypress''' in a separate terminal session, and use the locally provided command, '''seff <job-ID>''', to determine the the usage and efficiency of the job's CPU and memory.

{{{
[tulaneID@cypress1 ~]$seff 3336903
Job ID: 3336903
Cluster: cypress
User/Group: cbaribault/hpcstaff
State: RUNNING (exit code 0:0)
Cores: 20
CPU Utilized: 0:00:00
CPU Efficiency: 0.0% of 00:02:26 core-walltime
Memory Utilized: 0.00 GB
Memory Efficiency: 0.0% of requested per-CPU memory x 20
------------------------------------------------------------
}}}

 '''For workshop''' using only 2 requested cores

{{{
[tulaneID@cypress1 ~]$seff 3336904
Job ID: 3336904
Cluster: cypress
User/Group: cbaribault/hpcstaff
State: RUNNING (exit code 0:0)
Cores: 2
CPU Utilized: 0:00:00
CPU Efficiency: 0.0% of 00:00:27 core-walltime
Memory Utilized: 0.00 GB
Memory Efficiency: 0.0% of requested per-CPU memory x 2
------------------------------------------------------------
}}}

 These percentages are quite far from the ideal value, 100% - not very good usage of the node's 20 requested cores and memory (default=3200MB per core on Cypress).


=== Example 2: a running batch job using R requesting 1 node ===

==== Prepare sample R code ====

 For this and the following example, download and make a copy of the sample R code via the following.

{{{
[tulaneID@cypress1 ~]$git clone https://hidekiCCS:@bitbucket.org/hidekiCCS/hpc-workshop.git
[tulaneID@cypress1 ~]$cp -r hpc-workshop/R/* .
[tulaneID@cypress1 ~]$ls
bootstrap.R  bootstrap.sh  bootstrapWargs.R  bootstrapWargs.sh  myRscript.R  slurmscript1  slurmscript2
}}}

 For demonstration purposes, the downloaded and copied R script, '''bootstrap.R''' (see [wiki:cypress/R#PassingSLURMEnvironmentVariables here]), has been modified to run 1000000 (1M) samples rather than the original 10000 (10K) samples.

{{{
[tulaneID@cypress1 ~]$diff bootstrap.R hpc-workshop/R/bootstrap.R
10c10
< iterations <- 1000000# Number of iterations to run
---
> iterations <- 10000# Number of iterations to run
bootstrap.R  bootstrap.sh  bootstrapWargs.R  bootstrapWargs.sh  myRscript.R  slurmscript1  slurmscript2
}}}

==== Submit the test batch job ====

 The job script, '''bootstrap.sh''', is requesting 1 node and 16 cores.

{{{
[tulaneID@cypress1 ~]$grep cpus-per-task bootstrap.sh
#SBATCH --cpus-per-task=16      # Number of threads per task (OMP threads)
[tulaneID@cypress1 ~]$sbatch bootstrap.sh
Submitted batch job 3289740
[tulaneID@cypress1 ~]$squeue -u $USER
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
           3289740 workshop7        R tulaneID  R       0:00      1 cypress01-009
# note - the following result after many attempts of top with result only %CPU ~= 100.0
[tulaneID@cypress1 ~]$ssh cypress01-009 'top -b -n 1 -u $USER' | \
awk 'NR > 7 { sum_cpu += $9; sum_mem += $10 } \
END { print "Total %CPU:", sum_cpu; print "Total %MEM:", sum_mem }'
Total %CPU: 1556.6
Total %MEM: 3.3
}}}

==== Calculate core efficiency ====

 The resulting core efficiency is 

{{{
[tulaneID@cypress1 ~]$bc <<< "scale=2;(1556.3 / 100) / 16"
.97
}}}

 This is quite close to the ideal value, 1 - fairly good usage of the node's 16 requested cores.

=== Example 3: same R code requesting 2 nodes - 1 node unused ===

 The following uses the same the R sampling code as above (see [wiki:cypress/R#PassingSLURMEnvironmentVariables here]) requesting 16 cores and 2 nodes (--nodes - one of which is unused.

{{{
[tulaneID@cypress1 ~]$diff bootstrap.sh bootstrap2nodes.sh
7c7
< #SBATCH --nodes=1               # Number of Nodes
---
> #SBATCH --nodes=2               # Number of Nodes
[tulaneID@cypress1 ~]$sbatch bootstrap2nodes.sh
Submitted batch job 3289779
[tulaneID@cypress1 ~]$squeue -u $USER
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
           3289779 workshop7        R tulaneID  R       0:03      2 cypress01-[009-010]
# use the following to list job nodes separately
[tulandID@cypress1 R]$scontrol show hostname cypress01-[009-010]
cypress01-009
cypress01-010
[tulaneID@cypress1 ~]$ssh cypress01-009 'top -b -n 1 -u $USER' | \
awk 'NR > 7 { sum_cpu += $9; sum_mem += $10 } \
END { print "Total %CPU:", sum_cpu; print "Total %MEM:", sum_mem }'
Total %CPU: 1587.6
Total %MEM: 3.3
[tulaneID@cypress1 ~]$ssh cypress01-010 'top -b -n 1 -u $USER' | \
awk 'NR > 7 { sum_cpu += $9; sum_mem += $10 } \
END { print "Total %CPU:", sum_cpu; print "Total %MEM:", sum_mem }'
Total %CPU: 13.3
Total %MEM: 0
}}}

 The resulting core efficiency for each of the two requested nodes is 
{{{
[tulaneID@cypress1 ~]$bc <<< "scale=3; (1587.6 / 100) / 16"
.992
[tulaneID@cypress1 ~]$bc <<< "scale=3; (13.3 / 100) / 16"
.008
}}}
 Result:
 * On the first node, cypress01-009, usage is '''nearly ideal''' (.992 ~= 1.0).
 * On the second node, cypress01-010, usage is '''nearly non-existent''' (.008 ~= 0.0).

=== Running R on multiple nodes ===

 For information on how to run R code on multiple nodes on a SLURM cluster, see [wiki:/cypress/R#RunningRonmultiplenodes Running R on multiple nodes].