While your job is running - determining current core efficiency

Assumptions

• Request sufficient processor resources

For running jobs let's assume that you've requested sufficient processor resources via the following. (See man sbatch.)

Description	SBATCH options	Default value	Maximum value
# of nodes	-N, --nodes	Subject to -n and -c options	See SLURM (resource manager)
# of tasks	-n, --ntasks	1 task per node	20 * (# of nodes)
# of cores/CPUs/processors per tasks	-c, --cpus-per-task	1 core per task	20
total Random Access Memory (RAM)	--mem	3200MB per core	64/128/256GB
RAM per core	--mem-per-cpu	3200MB	" " "

• Your job's memory requirement may be greater than your (# cores) * (Total RAM)/20.
  • For example, your job may require only 10 cores but all of the RAM available on a node with 128GB of RAM.

  ...
  #SBATCH --ntasks=1
  #SBATCH --cpus-per-task=10  # Use only 10 cores
  #SBATCH --mem=128           # Use all memory on 128GB node
  ...

Current core efficiency for running jobs: (actual core usage) / (requested core allocation)

Example 1: an idev job for an idle interactive session

1. Start an idev interactive session.

[tulaneID@cypress1 ~]$idev --partition=centos7
Requesting 1 node(s)  task(s) to normal queue of centos7 partition
1 task(s)/node, 20 cpu(s)/task, 0 MIC device(s)/node
Time: 0 (hr) 60 (min).
0d 0h 60m
Submitted batch job 3336903
JOBID=3336903 begin on cypress01-121
--> Creating interactive terminal session (login) on node cypress01-121.
--> You have 0 (hr) 60 (min).
--> Assigned Host List : /tmp/idev_nodes_file_cbaribault
Last login: Fri Dec  5 14:02:37 2025 from cypress2.cm.cluster

For workshop using only 2 requested cores

[tulaneID@cypress1 ~]$idev --partition=workshop7 -c 2

2. Login to Cypress in a separate terminal session, and use the locally provided command, seff <job-ID>, to determine the the usage and efficiency of the job's CPU and memory.

[tulaneID@cypress1 ~]$seff 3336903
Job ID: 3336903
Cluster: cypress
User/Group: cbaribault/hpcstaff
State: RUNNING (exit code 0:0)
Cores: 20
CPU Utilized: 0:00:00
CPU Efficiency: 0.0% of 00:02:26 core-walltime
Memory Utilized: 0.00 GB
Memory Efficiency: 0.0% of requested per-CPU memory x 20
------------------------------------------------------------

For workshop using only 2 requested cores

[tulaneID@cypress1 ~]$seff 3336904
Job ID: 3336904
Cluster: cypress
User/Group: cbaribault/hpcstaff
State: RUNNING (exit code 0:0)
Cores: 2
CPU Utilized: 0:00:00
CPU Efficiency: 0.0% of 00:00:27 core-walltime
Memory Utilized: 0.00 GB
Memory Efficiency: 0.0% of requested per-CPU memory x 2
------------------------------------------------------------

These percentages are quite far from the ideal value, 100% - not very good usage of the node's 20 requested cores and memory (default=3200MB per core on Cypress).

Example 2: a running batch job using R requesting 1 node

Prepare sample R code

For this and the following example, download and make a copy of the sample R code via the following.

[tulaneID@cypress1 ~]$git clone https://hidekiCCS:@bitbucket.org/hidekiCCS/hpc-workshop.git
[tulaneID@cypress1 ~]$cp -r hpc-workshop/R/* .
[tulaneID@cypress1 ~]$ls
bootstrap.R  bootstrap.sh  bootstrapWargs.R  bootstrapWargs.sh  myRscript.R  slurmscript1  slurmscript2

For demonstration purposes, the downloaded and copied R script, bootstrap.R (see here), has been modified to run 1000000 (1M) samples rather than the original 10000 (10K) samples.

[tulaneID@cypress1 ~]$diff bootstrap.R hpc-workshop/R/bootstrap.R
10c10
< iterations <- 1000000# Number of iterations to run
---
> iterations <- 10000# Number of iterations to run
bootstrap.R  bootstrap.sh  bootstrapWargs.R  bootstrapWargs.sh  myRscript.R  slurmscript1  slurmscript2

Submit the test batch job

The job script, bootstrap.sh, is requesting 1 node and 16 cores.

[tulaneID@cypress1 ~]$grep cpus-per-task bootstrap.sh
#SBATCH --cpus-per-task=16      # Number of threads per task (OMP threads)
[tulaneID@cypress1 ~]$sbatch bootstrap.sh
Submitted batch job 3289740
[tulaneID@cypress1 ~]$squeue -u $USER
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
           3289740 workshop7        R tulaneID  R       0:00      1 cypress01-009
# note - the following result after many attempts of top with result only %CPU ~= 100.0
[tulaneID@cypress1 ~]$ssh cypress01-009 'top -b -n 1 -u $USER' | \
awk 'NR > 7 { sum_cpu += $9; sum_mem += $10 } \
END { print "Total %CPU:", sum_cpu; print "Total %MEM:", sum_mem }'
Total %CPU: 1556.6
Total %MEM: 3.3

Calculate core efficiency

The resulting core efficiency is

[tulaneID@cypress1 ~]$bc <<< "scale=2;(1556.3 / 100) / 16"
.97

This is quite close to the ideal value, 1 - fairly good usage of the node's 16 requested cores.

Example 3: same R code requesting 2 nodes - 1 node unused

The following uses the same the R sampling code as above (see here) requesting 16 cores and 2 nodes (--nodes - one of which is unused.

[tulaneID@cypress1 ~]$diff bootstrap.sh bootstrap2nodes.sh
7c7
< #SBATCH --nodes=1               # Number of Nodes
---
> #SBATCH --nodes=2               # Number of Nodes
[tulaneID@cypress1 ~]$sbatch bootstrap2nodes.sh
Submitted batch job 3289779
[tulaneID@cypress1 ~]$squeue -u $USER
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
           3289779 workshop7        R tulaneID  R       0:03      2 cypress01-[009-010]
# use the following to list job nodes separately
[tulandID@cypress1 R]$scontrol show hostname cypress01-[009-010]
cypress01-009
cypress01-010
[tulaneID@cypress1 ~]$ssh cypress01-009 'top -b -n 1 -u $USER' | \
awk 'NR > 7 { sum_cpu += $9; sum_mem += $10 } \
END { print "Total %CPU:", sum_cpu; print "Total %MEM:", sum_mem }'
Total %CPU: 1587.6
Total %MEM: 3.3
[tulaneID@cypress1 ~]$ssh cypress01-010 'top -b -n 1 -u $USER' | \
awk 'NR > 7 { sum_cpu += $9; sum_mem += $10 } \
END { print "Total %CPU:", sum_cpu; print "Total %MEM:", sum_mem }'
Total %CPU: 13.3
Total %MEM: 0

The resulting core efficiency for each of the two requested nodes is

[tulaneID@cypress1 ~]$bc <<< "scale=3; (1587.6 / 100) / 16"
.992
[tulaneID@cypress1 ~]$bc <<< "scale=3; (13.3 / 100) / 16"
.008

Result:

• On the first node, cypress01-009, usage is nearly ideal (.992 ~= 1.0).
• On the second node, cypress01-010, usage is nearly non-existent (.008 ~= 0.0).

Running R on multiple nodes

For information on how to run R code on multiple nodes on a SLURM cluster, see Running R on multiple nodes.