While your job is running - determining current core efficiency

Assumptions

• Request sufficient processor resources

For running jobs let's assume that you've requested sufficient processor resources via the following. (See man sbatch.)

Description	SBATCH options	Default value	Maximum value
# of nodes	-N, —nodes	Subject to -n and -c options	See SLURM (resource manager)
# of tasks	-n, —ntasks	1 task per node	20 * (# of nodes)
# of cores/CPUs/processors per tasks	-c, —cpus-per-task	1 core per task	20
total Random Access Memory (RAM)	—mem	1 core per task	64/128/256GB
RAM per core	—mem-per-cpu	3200MB	" " "

• Your job's memory requirement may be greater than your (# cores) * (Total RAM)/20.
  • For example, your job may require only 10 cores but all of the RAM available on a node with 128GB of RAM.

--ntasks=1

--cpus-per-task=10

--mem=128

Current core efficiency for running jobs: (actual core usage) / (requested core allocation)

Example 1: an idev job for an idle interactive session

1. Start an idev interactive session.

[tulaneID@cypress1 ~]$idev --partition=centos7
Requesting 1 node(s)  task(s) to normal queue of centos7 partition
1 task(s)/node, 20 cpu(s)/task, 0 MIC device(s)/node
Time: 0 (hr) 60 (min).
0d 0h 60m
Submitted batch job 3272175
JOBID=3289908 begin on cypress01-066
--> Creating interactive terminal session (login) on node cypress01-066.
--> You have 0 (hr) 60 (min).
--> Assigned Host List : /tmp/idev_nodes_file_tulaneID
Last login: Thu Jan 15 14:42:40 2026 from cypress2.cm.cluster

For workshop using only 2 requested cores

[tulaneID@cypress1 ~]$idev --partition=workshop7 -c 2

2. Login to Cypress in a separate terminal session, and use the SLURM squeue command to determine the job's node list - in this case an idle interactive session.

[tulaneID@cypress1 ~]$squeue -u $USER
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
           3272175   centos7 idv38428 tulaneID  R       3:54      1 cypress01-059

3. For each job node, use the ssh and top commands in combination to determine the job's core usage on the given node such as the following. (See man top.)

Here are the relevant output columns for the top command.

top command output column	Description	Notes
%CPU	percentage of cores used per job process (100% per full core used)	sum(%CPU)/100=fractional # of cores in use on the node
%MEM	percentage of RAM used per job process	sum(%MEM)=percentage of node's total RAM in use

Here's the combined command and result.

[tulaneID@cypress1 ~]$ssh cypress01-059 top -b -n 1 -u $USER
top - 00:34:01 up 75 days,  9:23,  1 user,  load average: 0.06, 0.05, 0.01
Tasks: 730 total,   1 running, 729 sleeping,   0 stopped,   0 zombie
Cpu(s): 52.7%us,  0.7%sy,  0.0%ni, 46.6%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  66013252k total,  3374764k used, 62638488k free,   137060k buffers
Swap: 12582904k total,        0k used, 12582904k free,  1280248k cached

   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 31502 tulaneID  20   0 27880 1772  956 R  3.8  0.0   0:00.09 top
 30927 tulaneID  20   0  9200 1244 1044 S  0.0  0.0   0:00.00 slurm_script
 30953 tulaneID  20   0  4072  544  464 S  0.0  0.0   0:00.00 sleep
 30970 tulaneID  20   0  144m 2368 1164 S  0.0  0.0   0:00.00 sshd
 30971 tulaneID  20   0 25092 3100 1516 S  0.0  0.0   0:00.06 bash
 31501 tulaneID  20   0  144m 2316 1132 S  0.0  0.0   0:00.00 sshd

4. Next we'll re-run the same combined ssh…top… command and pipe the input to awk in order to sum the values in the columns %CPU, %MEM.

[tulaneID@cypress1 ~]$ssh cypress01-065 'top -b -n 1 -u $USER' | \
   awk 'NR > 7 { sum_cpu += $9; sum_mem += $10 } \
   END { print "Total %CPU:", sum_cpu; print "Total %MEM:", sum_mem }'
Total %CPU: 3.8
Total %MEM: 0

5. If the idev session requested 20 cores (default=20), then the core efficiency of the idle idev session is

[tulaneID@cypress1 ~]$bc <<< "scale=4;(3.8 / 100) / 20"
.0019

This is quite far from the ideal value, 1 - not very good usage of the node's 20 requested cores.

For workshop using only 2 requested cores

[tulaneID@cypress1 ~]$bc <<< "scale=4;(3.8 / 100) / 2"
.0190

Example 2: a running batch job using R requesting 1 node

Prepare sample R code

For this and the following example, download and make a copy of the sample R code via the following.

[tulaneID@cypress1 ~]$git clone https://hidekiCCS:@bitbucket.org/hidekiCCS/hpc-workshop.git
[tulaneID@cypress1 ~]$cp -r hpc-workshop/R/* .
[tulaneID@cypress1 ~]$ls
bootstrap.R  bootstrap.sh  bootstrapWargs.R  bootstrapWargs.sh  myRscript.R  slurmscript1  slurmscript2

For demonstration purposes, the downloaded and copied R script, bootstrap.R (see here), has been modified to run 1000000 (1M) samples rather than the original 10000 (10K) samples.

[tulaneID@cypress1 ~]$diff bootstrap.R hpc-workshop/R/bootstrap.R
10c10
< iterations <- 1000000# Number of iterations to run
---
> iterations <- 10000# Number of iterations to run
bootstrap.R  bootstrap.sh  bootstrapWargs.R  bootstrapWargs.sh  myRscript.R  slurmscript1  slurmscript2

Submit the test batch job

The job script, bootstrap.sh, is requesting 1 node and 16 cores.

[tulaneID@cypress1 ~]$grep cpus-per-task bootstrap.sh
#SBATCH --cpus-per-task=16      # Number of threads per task (OMP threads)
[tulaneID@cypress1 ~]$sbatch bootstrap.sh
Submitted batch job 3289740
[tulaneID@cypress1 ~]$squeue -u $USER
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
           3289740 workshop7        R tulaneID  R       0:00      1 cypress01-009
# note - the following result after many attempts of top with result only %CPU ~= 100.0
[tulaneID@cypress1 ~]$ssh cypress01-009 'top -b -n 1 -u $USER' | \
awk 'NR > 7 { sum_cpu += $9; sum_mem += $10 } \
END { print "Total %CPU:", sum_cpu; print "Total %MEM:", sum_mem }'
Total %CPU: 1556.6
Total %MEM: 3.3

Calculate core efficiency

The resulting core efficiency is

[tulaneID@cypress1 ~]$bc <<< "scale=2;(1556.3 / 100) / 16"
.97

This is quite close to the ideal value, 1 - fairly good usage of the node's 16 requested cores.

Example 3: same R code requesting 2 nodes - 1 node unused

The following uses the same the R sampling code as above (see here) requesting 16 cores and 2 nodes (—nodes - one of which is unused.

[tulaneID@cypress1 ~]$diff bootstrap.sh bootstrap2nodes.sh
7c7
< #SBATCH --nodes=1               # Number of Nodes
---
> #SBATCH --nodes=2               # Number of Nodes
[tulaneID@cypress1 ~]$sbatch bootstrap2nodes.sh
Submitted batch job 3289779
[tulaneID@cypress1 ~]$squeue -u $USER
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
           3289779 workshop7        R tulaneID  R       0:03      2 cypress01-[009-010]
# use the following to list job nodes separately
[tulandID@cypress1 R]$scontrol show hostname cypress01-[009-010]
cypress01-009
cypress01-010
[tulaneID@cypress1 ~]$ssh cypress01-009 'top -b -n 1 -u $USER' | \
awk 'NR > 7 { sum_cpu += $9; sum_mem += $10 } \
END { print "Total %CPU:", sum_cpu; print "Total %MEM:", sum_mem }'
Total %CPU: 1587.6
Total %MEM: 3.3
[tulaneID@cypress1 ~]$ssh cypress01-010 'top -b -n 1 -u $USER' | \
awk 'NR > 7 { sum_cpu += $9; sum_mem += $10 } \
END { print "Total %CPU:", sum_cpu; print "Total %MEM:", sum_mem }'
Total %CPU: 13.3
Total %MEM: 0

The resulting core efficiency for each of the two requested nodes is

[tulaneID@cypress1 ~]$bc <<< "scale=3; (1587.6 / 100) / 16"
.992
[tulaneID@cypress1 ~]$bc <<< "scale=3; (13.3 / 100) / 16"
.008

Result:

• On the first node, cypress01-009, usage is nearly ideal (.992 ~= 1.0).
• On the second node, cypress01-010, usage is nearly non-existent (.008 ~= 0.0).

Running R on multiple nodes

For information on how to run R code on multiple nodes on a SLURM cluster, see Running R on multiple nodes.