Changes between Version 10 and Version 11 of Workshops/JobParallelism/WhileYourJobIsRunning

Timestamp:

09/02/2026 01:46:47 AM (43 hours ago)

Author:

Carl Baribault

Comment:

Removed multi-node for now and improved single-node example

Workshops/JobParallelism/WhileYourJobIsRunning

@@ -91 +91 @@
 === Example 2: a running batch job using R requesting 1 node ===
 
-==== Prepare sample R code ====
-
- For this and the following example, download and make a copy of the sample R code via the following.
-
-{{{
-[tulaneID@cypress1 ~]$git clone https://hidekiCCS:@bitbucket.org/hidekiCCS/hpc-workshop.git
-[tulaneID@cypress1 ~]$cp -r hpc-workshop/R/* .
-[tulaneID@cypress1 ~]$ls
-bootstrap.R  bootstrap.sh  bootstrapWargs.R  bootstrapWargs.sh  myRscript.R  slurmscript1  slurmscript2
-}}}
-
- For demonstration purposes, the downloaded and copied R script, '''bootstrap.R''' (see [wiki:cypress/R#PassingSLURMEnvironmentVariables here]), has been modified to run 1000000 (1M) samples rather than the original 10000 (10K) samples.
-
-{{{
-[tulaneID@cypress1 ~]$diff bootstrap.R hpc-workshop/R/bootstrap.R
-10c10
-< iterations <- 1000000# Number of iterations to run
----
-> iterations <- 10000# Number of iterations to run
-bootstrap.R  bootstrap.sh  bootstrapWargs.R  bootstrapWargs.sh  myRscript.R  slurmscript1  slurmscript2
-}}}
-
-==== Submit the test batch job ====
-
- The job script, '''bootstrap.sh''', is requesting 1 node and 16 cores.
-
-{{{
-[tulaneID@cypress1 ~]$grep cpus-per-task bootstrap.sh
+==== Prepare sample R script ====
+
+ For this example, we'll use the following R script in the file '''bootstrapFutureApply.R'''.
+
+ Note that we'll need to install the R package '''future.apply''' ahead of time.
+
+{{{
+[tulaneID@cypress1 ~]$cat bootstrapFutureApply.R
+library(future.apply)
+
+ncores <- as.integer(Sys.getenv("SLURM_CPUS_PER_TASK"))
+
+plan(multisession, workers = ncores)
+cat("Future plan =", class(plan())[1], "\n")
+
+cat("SLURM_CPUS_PER_TASK =", ncores, "\n")
+cat("Workers =", nbrOfWorkers(), "\n")
+
+x <- iris[which(iris[,5] != "setosa"), c(1,5)]
+
+iterations <- 100000
+
+chunks <- split(
+    seq_len(iterations),
+    cut(seq_len(iterations), ncores, labels = FALSE)
+)
+
+part <- system.time({
+    results <- future_lapply(
+        chunks,
+        function(idx) {
+            local_results <- vector("list", length(idx))
+            for (j in seq_along(idx)) {
+                ind <- sample(100, 100, replace = TRUE)
+                result1 <- glm(
+                    x[ind,2] ~ x[ind,1],
+                    family = binomial()
+                )
+
+                local_results[[j]] <- coefficients(result1)
+            }
+
+            do.call(cbind, local_results)
+        },
+        future.seed = TRUE
+    )
+
+    r <- do.call(cbind, results)
+
+})[3]
+
+print(part)
+plan(sequential)
+}}}
+
+In contrast with previous examples in [[cypress/R#RunningaRscriptinBatchmode|Running R with Batch Mode]], '''bootstrapFutureApply.R''' above uses future.apply to execute larger pre-assigned chunks of iterations per worker, reducing scheduling overhead but making workload balance and result aggregation more explicit.
+
+Whereas the previous '''bootstrap.R''' uses doParallel/foreach to dynamically distribute individual bootstrap iterations across worker processes with excellent CPU utilization yet with significant overhead spent in the function '''cbind''' for allocating and combining the results of the individual samples.
+ 
+==== Prepare sample job script ====
+
+Also, we'll use the following job script, '''bootstrapFutureApply.sh''', using the latest available version of R, 4.4.1, and is requesting 1 node and 16 cores for 1 hour.
+
+{{{
+[tulaneID@cypress1 ~]$cat bootstrapFutureApply.sh
+#!/bin/bash
+#SBATCH --qos=normal            # Quality of Service
+#SBATCH --partition=centos7     # Partition
+##SBATCH --qos=workshop         # Quality of Service
+##SBATCH --partition=workshop   # Partition
+#SBATCH --job-name=R            # Job Name
+#SBATCH --time=1:00:00          # WallTime
+#SBATCH --nodes=1               # Number of Nodes
+#SBATCH --ntasks-per-node=1     # Number of Tasks per Node
 #SBATCH --cpus-per-task=16      # Number of threads per task (OMP threads)
-[tulaneID@cypress1 ~]$sbatch bootstrap.sh
-Submitted batch job 3289740
-[tulaneID@cypress1 ~]$squeue -u $USER
-             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
-           3289740 workshop7        R tulaneID  R       0:00      1 cypress01-009
-# note - the following result after many attempts of top with result only %CPU ~= 100.0
-[tulaneID@cypress1 ~]$ssh cypress01-009 'top -b -n 1 -u $USER' | \
-awk 'NR > 7 { sum_cpu += $9; sum_mem += $10 } \
-END { print "Total %CPU:", sum_cpu; print "Total %MEM:", sum_mem }'
-Total %CPU: 1556.6
-Total %MEM: 3.3
-}}}
-
-==== Calculate core efficiency ====
-
- The resulting core efficiency is 
-
-{{{
-[tulaneID@cypress1 ~]$bc <<< "scale=2;(1556.3 / 100) / 16"
-.97
-}}}
-
- This is quite close to the ideal value, 1 - fairly good usage of the node's 16 requested cores.
-
-=== Example 3: same R code requesting 2 nodes - 1 node unused ===
-
- The following uses the same the R sampling code as above (see [wiki:cypress/R#PassingSLURMEnvironmentVariables here]) requesting 16 cores and 2 nodes (--nodes - one of which is unused.
-
-{{{
-[tulaneID@cypress1 ~]$diff bootstrap.sh bootstrap2nodes.sh
-7c7
-< #SBATCH --nodes=1               # Number of Nodes
----
-> #SBATCH --nodes=2               # Number of Nodes
-[tulaneID@cypress1 ~]$sbatch bootstrap2nodes.sh
-Submitted batch job 3289779
-[tulaneID@cypress1 ~]$squeue -u $USER
-             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
-           3289779 workshop7        R tulaneID  R       0:03      2 cypress01-[009-010]
-# use the following to list job nodes separately
-[tulandID@cypress1 R]$scontrol show hostname cypress01-[009-010]
-cypress01-009
-cypress01-010
-[tulaneID@cypress1 ~]$ssh cypress01-009 'top -b -n 1 -u $USER' | \
-awk 'NR > 7 { sum_cpu += $9; sum_mem += $10 } \
-END { print "Total %CPU:", sum_cpu; print "Total %MEM:", sum_mem }'
-Total %CPU: 1587.6
-Total %MEM: 3.3
-[tulaneID@cypress1 ~]$ssh cypress01-010 'top -b -n 1 -u $USER' | \
-awk 'NR > 7 { sum_cpu += $9; sum_mem += $10 } \
-END { print "Total %CPU:", sum_cpu; print "Total %MEM:", sum_mem }'
-Total %CPU: 13.3
-Total %MEM: 0
-}}}
-
- The resulting core efficiency for each of the two requested nodes is 
-{{{
-[tulaneID@cypress1 ~]$bc <<< "scale=3; (1587.6 / 100) / 16"
-.992
-[tulaneID@cypress1 ~]$bc <<< "scale=3; (13.3 / 100) / 16"
-.008
-}}}
- Result:
- * On the first node, cypress01-009, usage is '''nearly ideal''' (.992 ~= 1.0).
- * On the second node, cypress01-010, usage is '''nearly non-existent''' (.008 ~= 0.0).
-
-=== Running R on multiple nodes ===
-
- For information on how to run R code on multiple nodes on a SLURM cluster, see [wiki:/cypress/R#RunningRonmultiplenodes Running R on multiple nodes].
+
+module load R/4.4.1
+
+Rscript bootstrapFutureApply.R
+}}}
+
+ '''For workshop''' modify the above using the comment character '''#''' in order to use '''workshop''' and '''workshop7''' for '''qos''' and '''partition''', respectively
+
+{{{
+...
+##SBATCH --qos=normal            # Quality of Service
+##SBATCH --partition=centos7     # Partition
+#SBATCH --qos=workshop           # Quality of Service
+#SBATCH --partition=workshop7    # Partition
+...
+}}}
+
+==== Submit the test batch job and analyze usage ====
+
+{{{
+[tulaneID@cypress1 ~]$sbatch bootstrapFutureApply.sh
+Submitted batch job 3336943
+[tulaneID@cypress1 ~]$seff 3336943
+Job ID: 3336943
+Cluster: cypress
+User/Group: cbaribault/hpcstaff
+State: RUNNING (exit code 0:0)
+Cores: 16
+CPU Utilized: 0:00:00
+CPU Efficiency: 0.0% of 00:00:32 core-walltime
+Memory Utilized: 0.00 GB
+Memory Efficiency: 0.0% of requested per-CPU memory x 16
+------------------------------------------------------------
+}}}
+
+ Unfortunately the seff command is reports zero usage of core and memory due to a lack of information available on the running job in the SLURM job database.
+
+==== For a running job ====
+
+ Until we can enhance the local seff command, we can define and use a shell function, '''seffRunningPrototype''', shown further below as a workaround for now.
+
+{{{
+[tulaneID@cypress1 ~]$seffRunningPrototype 3336943
+JobID: 3336943
+Node: cypress01-058
+User: cbaribault
+Allocated CPUs: 16
+Total %CPU: 1531.7
+Total %MEM: 0.0
+Core efficiency: 0.96
+}}}
+
+ Here admittedly, the memory usage is not optimal, but the core efficiency is quite good at .96 - near the ideal 1.00.
+
+==== Correlate with results from top command ====
+
+ Here also is the result of running the top command on the running job's compute node, which correlates fairly well with the result from above.
+
+{{{
+[tulaneID@cypress1 ~]$ssh cypress01-058 top -b -n 1 -u cbaribault
+top - 01:19:36 up 5 days, 14:05,  0 users,  load average: 10.80, 3.41, 1.45
+Tasks: 472 total,  17 running, 455 sleeping,   0 stopped,   0 zombie
+%Cpu(s): 80.7 us,  0.3 sy,  0.0 ni, 19.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
+KiB Mem : 65879704 total, 60164396 free,  2180912 used,  3534396 buff/cache
+KiB Swap: 12582908 total, 12582908 free,        0 used. 62270792 avail Mem
+
+   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
+ 57357 cbariba+  20   0  241000  85116   7420 R 100.0  0.1   0:54.25 R
+ 57359 cbariba+  20   0  240856  84988   7420 R 100.0  0.1   0:54.41 R
+ 57360 cbariba+  20   0  241020  85076   7420 R 100.0  0.1   0:54.25 R
+ 57361 cbariba+  20   0  240996  85092   7420 R 100.0  0.1   0:54.15 R
+ 57362 cbariba+  20   0  240980  85020   7420 R 100.0  0.1   0:54.29 R
+ 57363 cbariba+  20   0  240860  85008   7420 R 100.0  0.1   0:54.12 R
+ 57364 cbariba+  20   0  240872  84984   7420 R 100.0  0.1   0:54.13 R
+ 57365 cbariba+  20   0  241008  85128   7420 R 100.0  0.1   0:54.12 R
+ 57366 cbariba+  20   0  241000  85172   7420 R 100.0  0.1   0:54.29 R
+ 57367 cbariba+  20   0  240872  85012   7420 R 100.0  0.1   0:54.12 R
+ 57369 cbariba+  20   0  240992  85040   7420 R 100.0  0.1   0:54.21 R
+ 57355 cbariba+  20   0  240988  84988   7420 R  93.8  0.1   0:54.13 R
+ 57356 cbariba+  20   0  241016  85124   7420 R  93.8  0.1   0:54.18 R
+ 57358 cbariba+  20   0  241132  84996   7420 R  93.8  0.1   0:54.18 R
+ 57368 cbariba+  20   0  240872  84976   7420 R  93.8  0.1   0:54.21 R
+ 57370 cbariba+  20   0  240972  85020   7420 R  93.8  0.1   0:54.15 R
+ 57307 cbariba+  20   0  237004  80980   7316 S   6.2  0.1   0:17.80 R
+ 58217 cbariba+  20   0   69372   2416   1508 R   6.2  0.0   0:00.03 top
+ 57278 cbariba+  20   0    9568   1356   1124 S   0.0  0.0   0:00.00 slurm_scr+
+ 58216 cbariba+  20   0  180444   2480   1128 S   0.0  0.0   0:00.00 sshd
+
+}}}
+
+
+ Here is the definition of '''seffRunningPrototype''' in the source file seffRunningPrototype.sh.
+
+{{{
+[tulaneID@cypress1 ~]$cat seffRunningPrototype.sh
+# CAVEAT:
+# This function measures ALL processes owned by the user on the node.
+# If the user has multiple Slurm jobs, interactive sessions, Jupyter
+# notebooks, or other processes on the same node, CPU and memory
+# utilization may be inflated.
+#
+# For true job-specific accounting, process ownership should be
+# restricted to the Slurm job's process tree (slurmstepd descendants).
+# Show live CPU and memory utilization for a running Slurm job.
+#
+# Notes:
+# - CPU utilization is gathered from all processes owned by the job's user
+#   on the allocated node.
+# - This assumes the user is not running multiple jobs on the same node.
+# - Core efficiency is computed as:
+#
+#       total_cpu_percent / (100 * allocated_cpus)
+#
+#   Examples:
+#       1600% CPU on a 16-core allocation = 1.00 efficiency
+#        800% CPU on a 16-core allocation = 0.50 efficiency
+#
+seffRunningPrototype()
+{
+    # Slurm job ID supplied by the caller.
+    local jobid="$1"
+
+    # Query Slurm for:
+    #   node  = allocated compute node
+    #   user  = job owner
+    #   cpus  = CPUs allocated to the job
+    read node user cpus <<< $(
+        squeue -j "$jobid" \
+               --noheader \
+               --format="%.100N %.30u %.10C"
+    )
+
+    # Ensure the job was found.
+    if [ -z "$node" ]; then
+        echo "Job $jobid not found."
+        return 1
+    fi
+
+    echo "JobID: $jobid"
+    echo "Node: $node"
+    echo "User: $user"
+    echo "Allocated CPUs: $cpus"
+
+    # Execute on the compute node:
+    #
+    #   ps -u <user>
+    #
+    # returns one line per process with:
+    #   %CPU %MEM
+    #
+    # AWK sums CPU and memory across all matching processes.
+    ssh "$node" "
+        ps -u $user -o pcpu=,pmem= |
+        awk -v cpus=$cpus '
+        {
+            cpu += \$1
+            mem += \$2
+        }
+        END {
+            printf \"Total %%CPU: %.1f\n\", cpu
+            printf \"Total %%MEM: %.1f\n\", mem
+            printf \"Core efficiency: %.2f\n\", cpu / (100 * cpus)
+        }'
+    "
+}
+}}}