Changes between Version 20 and Version 21 of cypress/using

Timestamp:

04/30/15 14:29:00 (10 years ago)

Author:

fuji

Comment:

fixed SLURM_JOB_CPUS_PER_NODE → SLURM_CPUS_PER_TASK

cypress/using

@@ -141 +141 @@
 #SBATCH --cpus-per-task=20
 
-export OMP_NUM_THREADS=$SLURM_JOB_CPUS_PER_NODE
+export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
 
 ./myOMPexecutable
@@ -147 +147 @@
 
 
-In the script above we request 20 cores on one node of Cypress (which is all the cores available on any node). As SLURM regards tasks as being analogous to MPI processes, it’s better to use the cpus-per-task directive when employing OpenMP parallelism. Additionally, the SLURM export variable $SLURM_JOB_CPUS_PER_NODE stores whatever value we assign to cpus-per-task, and is therefore our candidate for passing to OMP_NUM_THREADS.
+In the script above we request 20 cores on one node of Cypress (which is all the cores available on any node). As SLURM regards tasks as being analogous to MPI processes, it’s better to use the cpus-per-task directive when employing OpenMP parallelism. Additionally, the SLURM export variable $SLURM_CPUS_PER_TASK stores whatever value we assign to cpus-per-task, and is therefore our candidate for passing to OMP_NUM_THREADS.
+
+=== Hybrid Jobs ===
+
+When running MPI/OpenMP hybrid jobs on Cypress, for example,
+
+{{{#!bash
+#!/bin/bash
+#SBATCH --qos=normal            # Quality of Service
+#SBATCH --job-name=hybridTest   # Job Name
+#SBATCH --time=00:10:00         # WallTime
+#SBATCH --nodes=2               # Number of Nodes
+#SBATCH --ntasks-per-node=2     # Number of tasks (MPI processes)
+#SBATCH --cpus-per-task=10      # Number of threads per task (OMP threads)
+
+export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
+
+mpirun ./myHybridExecutable
+}}}
+
+In the script above we request 2 tasks per node and 10 cpus per task, which means 20 cores per node and all the cores available on one node.
+We request 2 nodes so that we can use 4 MPI processes. Each process can use 10 OpenMP threads.
+
+
+
+
+
+
+=== MIC Native Jobs ===
+
+There are two ''Intel Xeon Phi'' co-processors (MIC) on each node of Cypress. To run your code natively on MIC, make sure you have to compile the code with "-mmic" option.
+The executable for MIC code cannot run on host CPU. To launch the MIC native executable from host, use "micnativeloadex" command.
+SLURM jobscript script is, for example,
+
+{{{#!bash
+#!/bin/bash
+#SBATCH --qos=normal            # Quality of Service
+#SBATCH --job-name=nativeTest   # Job Name
+#SBATCH --time=00:10:00         # WallTime
+#SBATCH --nodes=1               # Number of Nodes
+#SBATCH --ntasks-per-node=1     # Number of tasks (MPI presseces)
+#SBATCH --cpus-per-task=1       # Number of processors per task OpenMP threads()
+#SBATCH --gres=mic:1            # Number of Co-Processors
+
+micnativeloadex ./myNativeExecutable -e "OMP_NUM_THREADS=100" -d 0 -v
+}}}
+
+
+
+In the script above we request one MIC device that will be device number 0. 
+"micnativeloadex" command launches MIC native executable. "-e "OMP_NUM_THREADS=100"" option to set the number of threads on the MIC device to 100.
+For more options, see below.
+
+{{{#!bash
+[fuji@cypress01-090 nativeTest]$ micnativeloadex -h
+
+Usage:
+micnativeloadex [ -h | -V ] AppName -l -t timeout -p -v -d coprocessor -a "args" -e "environment"
+  -a "args" An optional string of command line arguments to pass to
+            the remote app.
+  -d The (zero based) index of the Intel(R) Xeon Phi(TM) coprocessor to run the app on.
+  -e "environment" An optional environment string to pass to the remote app.
+      Multiple environment variable may be specified using spaces as separators:
+        -e "LD_LIBRARY_PATH=/lib64/ DEBUG=1"
+  -h Print this help message
+  -l Do not execute the binary on the coprocessor. Instead, list the shared library
+     dependency information.
+  -p Disable console proxy.
+  -t Time to wait for the remote app to finish (in seconds). After the timeout
+     is reached the remote app will be terminated.
+  -v Enable verbose mode. Note that verbose output will be displayed
+     if the remote app terminates abnormally.
+  -V Show version and build information
+}}}
+
+
+
+
+
 
 == Submitting an interactive job ==
@@ -182 +260 @@
 
 Note the prompt, "cypress01-100", in the above session. It is your interactive compute-node prompt. You can load modules, compile codes and test codes.
-`idev` transfers the environmental variables to computing node. Therefore, if you have loaded some module on the login node, you don't have to load the same module again.
+`idev` transfers the environmental variables to computing node. Therefore, if you have loaded some modules on the login node, you don't have to load the same module again.
 
 ==== Options ====
@@ -213 +291 @@
 }}}
 
+==== MIC native run ====
+
+You can login to MIC device and run native codes. There are two MIC devices on each node, mic0 and mic1.
+
+{{{#!bash
+[fuji@cypress01-100 nativeTest]$ ssh mic0
+fuji@cypress01-100-mic0:~$
+}}}
+
+The prompt, "cypress01-100-mic0" in the above session is your interactive MIC device prompt. Note that you cannot run CPU code there. Therefore, you cannot compile code on device, even compiling for native code.
+The environmental variables are not set and also ''module'' command doesn't work. To run your native executable that uses shared libraries, you have to set environmental variables manually, like
+{{{#!bash
+export LD_LIBRARY_PATH=/share/apps/intel_parallel_studio_xe/2015_update1/lib/mic:$LD_LIBRARY_PATH
+}}}
+
 ==== Questions or Suggestions ====