wiki:Workshops/JobParallelism/WhileYourJobIsRunning

Context Navigation

Version 11 (modified by Carl Baribault, 43 hours ago) ( diff )
Removed multi-node for now and improved single-node example

While your job is running - determining current core efficiency
1. Assumptions
2. Current core efficiency for running jobs: (actual core usage) / …
  1. Example 1: an idev job for an idle interactive session
  2. Example 2: a running batch job using R requesting 1 node

While your job is running - determining current core efficiency

Assumptions

Request sufficient processor resources

For running jobs let's assume that you've requested sufficient processor resources via the following. (See man sbatch.)

Description SBATCH options Default value Maximum value
# of nodes -N, --nodes Subject to -n and -c options See SLURM (resource manager)
# of tasks -n, --ntasks 1 task per node 20 * (# of nodes)
# of cores/CPUs/processors per tasks -c, --cpus-per-task 1 core per task 20
total Random Access Memory (RAM) --mem 3200MB per core 64/128/256GB
RAM per core --mem-per-cpu 3200MB " " "

Your job's memory requirement may be greater than your (# cores) * (Total RAM)/20.
- For example, your job may require only 10 cores but all of the RAM available on a node with 128GB of RAM.

  ...
  #SBATCH --ntasks=1
  #SBATCH --cpus-per-task=10  # Use only 10 cores
  #SBATCH --mem=128           # Use all memory on 128GB node
  ...

Current core efficiency for running jobs: (actual core usage) / (requested core allocation)

Example 1: an idev job for an idle interactive session

Start an idev interactive session.

[tulaneID@cypress1 ~]$idev --partition=centos7
Requesting 1 node(s)  task(s) to normal queue of centos7 partition
1 task(s)/node, 20 cpu(s)/task, 0 MIC device(s)/node
Time: 0 (hr) 60 (min).
0d 0h 60m
Submitted batch job 3336903
JOBID=3336903 begin on cypress01-121
--> Creating interactive terminal session (login) on node cypress01-121.
--> You have 0 (hr) 60 (min).
--> Assigned Host List : /tmp/idev_nodes_file_cbaribault
Last login: Fri Dec  5 14:02:37 2025 from cypress2.cm.cluster

For workshop using only 2 requested cores

[tulaneID@cypress1 ~]$idev --partition=workshop7 -c 2

Login to Cypress in a separate terminal session, and use the locally provided command, seff <job-ID>, to determine the the usage and efficiency of the job's CPU and memory.

[tulaneID@cypress1 ~]$seff 3336903
Job ID: 3336903
Cluster: cypress
User/Group: cbaribault/hpcstaff
State: RUNNING (exit code 0:0)
Cores: 20
CPU Utilized: 0:00:00
CPU Efficiency: 0.0% of 00:02:26 core-walltime
Memory Utilized: 0.00 GB
Memory Efficiency: 0.0% of requested per-CPU memory x 20
------------------------------------------------------------

For workshop using only 2 requested cores

[tulaneID@cypress1 ~]$seff 3336904
Job ID: 3336904
Cluster: cypress
User/Group: cbaribault/hpcstaff
State: RUNNING (exit code 0:0)
Cores: 2
CPU Utilized: 0:00:00
CPU Efficiency: 0.0% of 00:00:27 core-walltime
Memory Utilized: 0.00 GB
Memory Efficiency: 0.0% of requested per-CPU memory x 2
------------------------------------------------------------

These percentages are quite far from the ideal value, 100% - not very good usage of the node's 20 requested cores and memory (default=3200MB per core on Cypress).

Example 2: a running batch job using R requesting 1 node

Prepare sample R script

For this example, we'll use the following R script in the file bootstrapFutureApply.R.

Note that we'll need to install the R package future.apply ahead of time.

[tulaneID@cypress1 ~]$cat bootstrapFutureApply.R
library(future.apply)

ncores <- as.integer(Sys.getenv("SLURM_CPUS_PER_TASK"))

plan(multisession, workers = ncores)
cat("Future plan =", class(plan())[1], "\n")

cat("SLURM_CPUS_PER_TASK =", ncores, "\n")
cat("Workers =", nbrOfWorkers(), "\n")

x <- iris[which(iris[,5] != "setosa"), c(1,5)]

iterations <- 100000

chunks <- split(
    seq_len(iterations),
    cut(seq_len(iterations), ncores, labels = FALSE)
)

part <- system.time({
    results <- future_lapply(
        chunks,
        function(idx) {
            local_results <- vector("list", length(idx))
            for (j in seq_along(idx)) {
                ind <- sample(100, 100, replace = TRUE)
                result1 <- glm(
                    x[ind,2] ~ x[ind,1],
                    family = binomial()
                )

                local_results[[j]] <- coefficients(result1)
            }

            do.call(cbind, local_results)
        },
        future.seed = TRUE
    )

    r <- do.call(cbind, results)

})[3]

print(part)
plan(sequential)

In contrast with previous examples in Running R with Batch Mode, bootstrapFutureApply.R above uses future.apply to execute larger pre-assigned chunks of iterations per worker, reducing scheduling overhead but making workload balance and result aggregation more explicit.

Whereas the previous bootstrap.R uses doParallel/foreach to dynamically distribute individual bootstrap iterations across worker processes with excellent CPU utilization yet with significant overhead spent in the function cbind for allocating and combining the results of the individual samples.

Prepare sample job script

Also, we'll use the following job script, bootstrapFutureApply.sh, using the latest available version of R, 4.4.1, and is requesting 1 node and 16 cores for 1 hour.

[tulaneID@cypress1 ~]$cat bootstrapFutureApply.sh
#!/bin/bash
#SBATCH --qos=normal            # Quality of Service
#SBATCH --partition=centos7     # Partition
##SBATCH --qos=workshop         # Quality of Service
##SBATCH --partition=workshop   # Partition
#SBATCH --job-name=R            # Job Name
#SBATCH --time=1:00:00          # WallTime
#SBATCH --nodes=1               # Number of Nodes
#SBATCH --ntasks-per-node=1     # Number of Tasks per Node
#SBATCH --cpus-per-task=16      # Number of threads per task (OMP threads)

module load R/4.4.1

Rscript bootstrapFutureApply.R

For workshop modify the above using the comment character # in order to use workshop and workshop7 for qos and partition, respectively

...
##SBATCH --qos=normal            # Quality of Service
##SBATCH --partition=centos7     # Partition
#SBATCH --qos=workshop           # Quality of Service
#SBATCH --partition=workshop7    # Partition
...

Submit the test batch job and analyze usage

[tulaneID@cypress1 ~]$sbatch bootstrapFutureApply.sh
Submitted batch job 3336943
[tulaneID@cypress1 ~]$seff 3336943
Job ID: 3336943
Cluster: cypress
User/Group: cbaribault/hpcstaff
State: RUNNING (exit code 0:0)
Cores: 16
CPU Utilized: 0:00:00
CPU Efficiency: 0.0% of 00:00:32 core-walltime
Memory Utilized: 0.00 GB
Memory Efficiency: 0.0% of requested per-CPU memory x 16
------------------------------------------------------------

Unfortunately the seff command is reports zero usage of core and memory due to a lack of information available on the running job in the SLURM job database.

For a running job

Until we can enhance the local seff command, we can define and use a shell function, seffRunningPrototype, shown further below as a workaround for now.

[tulaneID@cypress1 ~]$seffRunningPrototype 3336943
JobID: 3336943
Node: cypress01-058
User: cbaribault
Allocated CPUs: 16
Total %CPU: 1531.7
Total %MEM: 0.0
Core efficiency: 0.96

Here admittedly, the memory usage is not optimal, but the core efficiency is quite good at .96 - near the ideal 1.00.

Correlate with results from top command

Here also is the result of running the top command on the running job's compute node, which correlates fairly well with the result from above.

[tulaneID@cypress1 ~]$ssh cypress01-058 top -b -n 1 -u cbaribault
top - 01:19:36 up 5 days, 14:05,  0 users,  load average: 10.80, 3.41, 1.45
Tasks: 472 total,  17 running, 455 sleeping,   0 stopped,   0 zombie
%Cpu(s): 80.7 us,  0.3 sy,  0.0 ni, 19.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 65879704 total, 60164396 free,  2180912 used,  3534396 buff/cache
KiB Swap: 12582908 total, 12582908 free,        0 used. 62270792 avail Mem

   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 57357 cbariba+  20   0  241000  85116   7420 R 100.0  0.1   0:54.25 R
 57359 cbariba+  20   0  240856  84988   7420 R 100.0  0.1   0:54.41 R
 57360 cbariba+  20   0  241020  85076   7420 R 100.0  0.1   0:54.25 R
 57361 cbariba+  20   0  240996  85092   7420 R 100.0  0.1   0:54.15 R
 57362 cbariba+  20   0  240980  85020   7420 R 100.0  0.1   0:54.29 R
 57363 cbariba+  20   0  240860  85008   7420 R 100.0  0.1   0:54.12 R
 57364 cbariba+  20   0  240872  84984   7420 R 100.0  0.1   0:54.13 R
 57365 cbariba+  20   0  241008  85128   7420 R 100.0  0.1   0:54.12 R
 57366 cbariba+  20   0  241000  85172   7420 R 100.0  0.1   0:54.29 R
 57367 cbariba+  20   0  240872  85012   7420 R 100.0  0.1   0:54.12 R
 57369 cbariba+  20   0  240992  85040   7420 R 100.0  0.1   0:54.21 R
 57355 cbariba+  20   0  240988  84988   7420 R  93.8  0.1   0:54.13 R
 57356 cbariba+  20   0  241016  85124   7420 R  93.8  0.1   0:54.18 R
 57358 cbariba+  20   0  241132  84996   7420 R  93.8  0.1   0:54.18 R
 57368 cbariba+  20   0  240872  84976   7420 R  93.8  0.1   0:54.21 R
 57370 cbariba+  20   0  240972  85020   7420 R  93.8  0.1   0:54.15 R
 57307 cbariba+  20   0  237004  80980   7316 S   6.2  0.1   0:17.80 R
 58217 cbariba+  20   0   69372   2416   1508 R   6.2  0.0   0:00.03 top
 57278 cbariba+  20   0    9568   1356   1124 S   0.0  0.0   0:00.00 slurm_scr+
 58216 cbariba+  20   0  180444   2480   1128 S   0.0  0.0   0:00.00 sshd

Here is the definition of seffRunningPrototype in the source file seffRunningPrototype.sh.

[tulaneID@cypress1 ~]$cat seffRunningPrototype.sh
# CAVEAT:
# This function measures ALL processes owned by the user on the node.
# If the user has multiple Slurm jobs, interactive sessions, Jupyter
# notebooks, or other processes on the same node, CPU and memory
# utilization may be inflated.
#
# For true job-specific accounting, process ownership should be
# restricted to the Slurm job's process tree (slurmstepd descendants).
# Show live CPU and memory utilization for a running Slurm job.
#
# Notes:
# - CPU utilization is gathered from all processes owned by the job's user
#   on the allocated node.
# - This assumes the user is not running multiple jobs on the same node.
# - Core efficiency is computed as:
#
#       total_cpu_percent / (100 * allocated_cpus)
#
#   Examples:
#       1600% CPU on a 16-core allocation = 1.00 efficiency
#        800% CPU on a 16-core allocation = 0.50 efficiency
#
seffRunningPrototype()
{
    # Slurm job ID supplied by the caller.
    local jobid="$1"

    # Query Slurm for:
    #   node  = allocated compute node
    #   user  = job owner
    #   cpus  = CPUs allocated to the job
    read node user cpus <<< $(
        squeue -j "$jobid" \
               --noheader \
               --format="%.100N %.30u %.10C"
    )

    # Ensure the job was found.
    if [ -z "$node" ]; then
        echo "Job $jobid not found."
        return 1
    fi

    echo "JobID: $jobid"
    echo "Node: $node"
    echo "User: $user"
    echo "Allocated CPUs: $cpus"

    # Execute on the compute node:
    #
    #   ps -u <user>
    #
    # returns one line per process with:
    #   %CPU %MEM
    #
    # AWK sums CPU and memory across all matching processes.
    ssh "$node" "
        ps -u $user -o pcpu=,pmem= |
        awk -v cpus=$cpus '
        {
            cpu += \$1
            mem += \$2
        }
        END {
            printf \"Total %%CPU: %.1f\n\", cpu
            printf \"Total %%MEM: %.1f\n\", mem
            printf \"Core efficiency: %.2f\n\", cpu / (100 * cpus)
        }'
    "
}

Note: See TracWiki for help on using the wiki.

Download in other formats:

Plain Text

Description	SBATCH options	Default value	Maximum value
# of nodes	-N, --nodes	Subject to -n and -c options	See SLURM (resource manager)
# of tasks	-n, --ntasks	1 task per node	20 * (# of nodes)
# of cores/CPUs/processors per tasks	-c, --cpus-per-task	1 core per task	20
total Random Access Memory (RAM)	--mem	3200MB per core	64/128/256GB
RAM per core	--mem-per-cpu	3200MB	" " "