[[PageOutline]] = While your job is running - determining current resource efficiency = == Assumptions == * Request sufficient processor resources For running jobs let's assume that you've requested sufficient processor resources via the following. (See '''man sbatch'''.) [[br]] ||='''Description'''=||='''SBATCH options'''=||='''Default value'''=||='''Maximum value'''=|| || # of nodes || -N, --nodes || Subject to -n and -c options || See [wiki:cypress/about#SLURMresourcemanager SLURM (resource manager)]|| || # of tasks || -n, --ntasks || 1 task per node || 20 * (# of nodes) || || # of cores/CPUs/processors per tasks || -c, --cpus-per-task || 1 core per task || 20 || || total Random Access Memory (RAM) || --mem || 3200MB per core || 64/128/256GB || || RAM per core || --mem-per-cpu || 3200MB || " " " || * Your job's memory requirement may be greater than your (# cores) * (Total RAM)/20. * For example, your job may require only 10 cores but all of the RAM available on a node with 128GB of RAM. {{{ ... #SBATCH --ntasks=1 #SBATCH --cpus-per-task=10 # Use only 10 cores #SBATCH --mem=128 # Use all memory on 128GB node ... }}} == Current core efficiency for running jobs: (actual core usage) / (requested core allocation) === Example 1: an idev job for an idle interactive session === 1. '''Start an idev interactive session.''' {{{ [tulaneID@cypress1 ~]$idev --partition=centos7 Requesting 1 node(s) task(s) to normal queue of centos7 partition 1 task(s)/node, 20 cpu(s)/task, 0 MIC device(s)/node Time: 0 (hr) 60 (min). 0d 0h 60m Submitted batch job 3336903 JOBID=3336903 begin on cypress01-121 --> Creating interactive terminal session (login) on node cypress01-121. --> You have 0 (hr) 60 (min). --> Assigned Host List : /tmp/idev_nodes_file_cbaribault Last login: Fri Dec 5 14:02:37 2025 from cypress2.cm.cluster }}} '''For workshop''' using only 2 requested cores {{{ [tulaneID@cypress1 ~]$idev --partition=workshop7 -c 2 }}} 2. '''Login to Cypress''' in a separate terminal session, and use the locally provided command, '''seff ''', to determine the the usage and efficiency of the job's CPU and memory. {{{ [tulaneID@cypress1 ~]$seff 3336903 Job ID: 3336903 Cluster: cypress User/Group: cbaribault/hpcstaff State: RUNNING (exit code 0:0) Cores: 20 CPU Utilized: 0:00:00 CPU Efficiency: 0.0% of 00:02:26 core-walltime Memory Utilized: 0.00 GB Memory Efficiency: 0.0% of requested per-CPU memory x 20 ------------------------------------------------------------ }}} '''For workshop''' using only 2 requested cores {{{ [tulaneID@cypress1 ~]$seff 3336904 Job ID: 3336904 Cluster: cypress User/Group: cbaribault/hpcstaff State: RUNNING (exit code 0:0) Cores: 2 CPU Utilized: 0:00:00 CPU Efficiency: 0.0% of 00:00:27 core-walltime Memory Utilized: 0.00 GB Memory Efficiency: 0.0% of requested per-CPU memory x 2 ------------------------------------------------------------ }}} These percentages are quite far from the ideal value, 100% - not very good usage of the node's 20 requested cores and memory (default=3200MB per core on Cypress). === Example 2: a running batch job using R requesting 1 node === ==== Prepare sample R script ==== For this example, we'll use the following R script in the file '''bootstrapFutureApply.R'''. Note that we'll need to install the R package '''future.apply''' ahead of time. {{{ [tulaneID@cypress1 ~]$cat bootstrapFutureApply.R library(future.apply) ncores <- as.integer(Sys.getenv("SLURM_CPUS_PER_TASK")) plan(multisession, workers = ncores) cat("Future plan =", class(plan())[1], "\n") cat("SLURM_CPUS_PER_TASK =", ncores, "\n") cat("Workers =", nbrOfWorkers(), "\n") x <- iris[which(iris[,5] != "setosa"), c(1,5)] iterations <- 100000 chunks <- split( seq_len(iterations), cut(seq_len(iterations), ncores, labels = FALSE) ) part <- system.time({ results <- future_lapply( chunks, function(idx) { local_results <- vector("list", length(idx)) for (j in seq_along(idx)) { ind <- sample(100, 100, replace = TRUE) result1 <- glm( x[ind,2] ~ x[ind,1], family = binomial() ) local_results[[j]] <- coefficients(result1) } do.call(cbind, local_results) }, future.seed = TRUE ) r <- do.call(cbind, results) })[3] print(part) plan(sequential) }}} In contrast with previous examples in [[cypress/R#RunningaRscriptinBatchmode|Running R with Batch Mode]], '''bootstrapFutureApply.R''' above uses future.apply to execute larger pre-assigned chunks of iterations per worker, reducing scheduling overhead but making workload balance and result aggregation more explicit. Whereas the previous '''bootstrap.R''' uses doParallel/foreach to dynamically distribute individual bootstrap iterations across worker processes with excellent CPU utilization yet with significant overhead spent in the function '''cbind''' for allocating and combining the results of the individual samples. ==== Prepare sample job script ==== Also, we'll use the following job script, '''bootstrapFutureApply.sh''', using the latest available version of R, 4.4.1, and is requesting 1 node and 16 cores for 1 hour. {{{ [tulaneID@cypress1 ~]$cat bootstrapFutureApply.sh #!/bin/bash #SBATCH --qos=normal # Quality of Service #SBATCH --partition=centos7 # Partition ##SBATCH --qos=workshop # Quality of Service ##SBATCH --partition=workshop # Partition #SBATCH --job-name=R # Job Name #SBATCH --time=1:00:00 # WallTime #SBATCH --nodes=1 # Number of Nodes #SBATCH --ntasks-per-node=1 # Number of Tasks per Node #SBATCH --cpus-per-task=16 # Number of threads per task (OMP threads) module load R/4.4.1 Rscript bootstrapFutureApply.R }}} '''For workshop''' modify the above using the comment character '''#''' in order to use '''workshop''' and '''workshop7''' for '''qos''' and '''partition''', respectively {{{ ... ##SBATCH --qos=normal # Quality of Service ##SBATCH --partition=centos7 # Partition #SBATCH --qos=workshop # Quality of Service #SBATCH --partition=workshop7 # Partition ... }}} ==== Submit the test batch job and analyze usage ==== {{{ [tulaneID@cypress1 ~]$sbatch bootstrapFutureApply.sh Submitted batch job 3336943 [tulaneID@cypress1 ~]$seff 3336943 Job ID: 3336943 Cluster: cypress User/Group: cbaribault/hpcstaff State: RUNNING (exit code 0:0) Cores: 16 CPU Utilized: 0:00:00 CPU Efficiency: 0.0% of 00:00:32 core-walltime Memory Utilized: 0.00 GB Memory Efficiency: 0.0% of requested per-CPU memory x 16 ------------------------------------------------------------ }}} Unfortunately the seff command is reports zero usage of core and memory due to a lack of information available on the running job in the SLURM job database. ==== For a running job ==== Until we can enhance the local seff command, we can define and use a shell function, '''seffRunningPrototype''', shown further below as a workaround for now. {{{ [tulaneID@cypress1 ~]$seffRunningPrototype 3336943 JobID: 3336943 Node: cypress01-058 User: cbaribault Allocated CPUs: 16 Total %CPU: 1531.7 Total %MEM: 0.0 Core efficiency: 0.96 }}} Here admittedly, the memory usage is not optimal, but the core efficiency is quite good at .96 - near the ideal 1.00. ==== Correlate with results from top command ==== Here also is the result of running the top command on the running job's compute node, which correlates fairly well with the result from above. {{{ [tulaneID@cypress1 ~]$ssh cypress01-058 top -b -n 1 -u cbaribault top - 01:19:36 up 5 days, 14:05, 0 users, load average: 10.80, 3.41, 1.45 Tasks: 472 total, 17 running, 455 sleeping, 0 stopped, 0 zombie %Cpu(s): 80.7 us, 0.3 sy, 0.0 ni, 19.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem : 65879704 total, 60164396 free, 2180912 used, 3534396 buff/cache KiB Swap: 12582908 total, 12582908 free, 0 used. 62270792 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 57357 cbariba+ 20 0 241000 85116 7420 R 100.0 0.1 0:54.25 R 57359 cbariba+ 20 0 240856 84988 7420 R 100.0 0.1 0:54.41 R 57360 cbariba+ 20 0 241020 85076 7420 R 100.0 0.1 0:54.25 R 57361 cbariba+ 20 0 240996 85092 7420 R 100.0 0.1 0:54.15 R 57362 cbariba+ 20 0 240980 85020 7420 R 100.0 0.1 0:54.29 R 57363 cbariba+ 20 0 240860 85008 7420 R 100.0 0.1 0:54.12 R 57364 cbariba+ 20 0 240872 84984 7420 R 100.0 0.1 0:54.13 R 57365 cbariba+ 20 0 241008 85128 7420 R 100.0 0.1 0:54.12 R 57366 cbariba+ 20 0 241000 85172 7420 R 100.0 0.1 0:54.29 R 57367 cbariba+ 20 0 240872 85012 7420 R 100.0 0.1 0:54.12 R 57369 cbariba+ 20 0 240992 85040 7420 R 100.0 0.1 0:54.21 R 57355 cbariba+ 20 0 240988 84988 7420 R 93.8 0.1 0:54.13 R 57356 cbariba+ 20 0 241016 85124 7420 R 93.8 0.1 0:54.18 R 57358 cbariba+ 20 0 241132 84996 7420 R 93.8 0.1 0:54.18 R 57368 cbariba+ 20 0 240872 84976 7420 R 93.8 0.1 0:54.21 R 57370 cbariba+ 20 0 240972 85020 7420 R 93.8 0.1 0:54.15 R 57307 cbariba+ 20 0 237004 80980 7316 S 6.2 0.1 0:17.80 R 58217 cbariba+ 20 0 69372 2416 1508 R 6.2 0.0 0:00.03 top 57278 cbariba+ 20 0 9568 1356 1124 S 0.0 0.0 0:00.00 slurm_scr+ 58216 cbariba+ 20 0 180444 2480 1128 S 0.0 0.0 0:00.00 sshd }}} Here is the definition of '''seffRunningPrototype''' in the source file seffRunningPrototype.sh. {{{ [tulaneID@cypress1 ~]$cat seffRunningPrototype.sh # CAVEAT: # This function measures ALL processes owned by the user on the node. # If the user has multiple Slurm jobs, interactive sessions, Jupyter # notebooks, or other processes on the same node, CPU and memory # utilization may be inflated. # # For true job-specific accounting, process ownership should be # restricted to the Slurm job's process tree (slurmstepd descendants). # Show live CPU and memory utilization for a running Slurm job. # # Notes: # - CPU utilization is gathered from all processes owned by the job's user # on the allocated node. # - This assumes the user is not running multiple jobs on the same node. # - Core efficiency is computed as: # # total_cpu_percent / (100 * allocated_cpus) # # Examples: # 1600% CPU on a 16-core allocation = 1.00 efficiency # 800% CPU on a 16-core allocation = 0.50 efficiency # seffRunningPrototype() { # Slurm job ID supplied by the caller. local jobid="$1" # Query Slurm for: # node = allocated compute node # user = job owner # cpus = CPUs allocated to the job read node user cpus <<< $( squeue -j "$jobid" \ --noheader \ --format="%.100N %.30u %.10C" ) # Ensure the job was found. if [ -z "$node" ]; then echo "Job $jobid not found." return 1 fi echo "JobID: $jobid" echo "Node: $node" echo "User: $user" echo "Allocated CPUs: $cpus" # Execute on the compute node: # # ps -u # # returns one line per process with: # %CPU %MEM # # AWK sums CPU and memory across all matching processes. ssh "$node" " ps -u $user -o pcpu=,pmem= | awk -v cpus=$cpus ' { cpu += \$1 mem += \$2 } END { printf \"Total %%CPU: %.1f\n\", cpu printf \"Total %%MEM: %.1f\n\", mem printf \"Core efficiency: %.2f\n\", cpu / (100 * cpus) }' " } }}}