Changes between Version 20 and Version 21 of cypress/using


Ignore:
Timestamp:
Apr 30, 2015 2:29:00 PM (7 years ago)
Author:
fuji
Comment:

fixed SLURM_JOB_CPUS_PER_NODE → SLURM_CPUS_PER_TASK

Legend:

Unmodified
Added
Removed
Modified
  • cypress/using

    v20 v21  
    141141#SBATCH --cpus-per-task=20
    142142
    143 export OMP_NUM_THREADS=$SLURM_JOB_CPUS_PER_NODE
     143export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
    144144
    145145./myOMPexecutable
     
    147147
    148148
    149 In the script above we request 20 cores on one node of Cypress (which is all the cores available on any node). As SLURM regards tasks as being analogous to MPI processes, it’s better to use the cpus-per-task directive when employing OpenMP parallelism. Additionally, the SLURM export variable $SLURM_JOB_CPUS_PER_NODE stores whatever value we assign to cpus-per-task, and is therefore our candidate for passing to OMP_NUM_THREADS.
     149In the script above we request 20 cores on one node of Cypress (which is all the cores available on any node). As SLURM regards tasks as being analogous to MPI processes, it’s better to use the cpus-per-task directive when employing OpenMP parallelism. Additionally, the SLURM export variable $SLURM_CPUS_PER_TASK stores whatever value we assign to cpus-per-task, and is therefore our candidate for passing to OMP_NUM_THREADS.
     150
     151=== Hybrid Jobs ===
     152
     153When running MPI/OpenMP hybrid jobs on Cypress, for example,
     154
     155{{{#!bash
     156#!/bin/bash
     157#SBATCH --qos=normal            # Quality of Service
     158#SBATCH --job-name=hybridTest   # Job Name
     159#SBATCH --time=00:10:00         # WallTime
     160#SBATCH --nodes=2               # Number of Nodes
     161#SBATCH --ntasks-per-node=2     # Number of tasks (MPI processes)
     162#SBATCH --cpus-per-task=10      # Number of threads per task (OMP threads)
     163
     164export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
     165
     166mpirun ./myHybridExecutable
     167}}}
     168
     169In the script above we request 2 tasks per node and 10 cpus per task, which means 20 cores per node and all the cores available on one node.
     170We request 2 nodes so that we can use 4 MPI processes. Each process can use 10 OpenMP threads.
     171
     172
     173
     174
     175
     176
     177=== MIC Native Jobs ===
     178
     179There are two ''Intel Xeon Phi'' co-processors (MIC) on each node of Cypress. To run your code natively on MIC, make sure you have to compile the code with "-mmic" option.
     180The executable for MIC code cannot run on host CPU. To launch the MIC native executable from host, use "micnativeloadex" command.
     181SLURM jobscript script is, for example,
     182
     183{{{#!bash
     184#!/bin/bash
     185#SBATCH --qos=normal            # Quality of Service
     186#SBATCH --job-name=nativeTest   # Job Name
     187#SBATCH --time=00:10:00         # WallTime
     188#SBATCH --nodes=1               # Number of Nodes
     189#SBATCH --ntasks-per-node=1     # Number of tasks (MPI presseces)
     190#SBATCH --cpus-per-task=1       # Number of processors per task OpenMP threads()
     191#SBATCH --gres=mic:1            # Number of Co-Processors
     192
     193micnativeloadex ./myNativeExecutable -e "OMP_NUM_THREADS=100" -d 0 -v
     194}}}
     195
     196
     197
     198In the script above we request one MIC device that will be device number 0.
     199"micnativeloadex" command launches MIC native executable. "-e "OMP_NUM_THREADS=100"" option to set the number of threads on the MIC device to 100.
     200For more options, see below.
     201
     202{{{#!bash
     203[fuji@cypress01-090 nativeTest]$ micnativeloadex -h
     204
     205Usage:
     206micnativeloadex [ -h | -V ] AppName -l -t timeout -p -v -d coprocessor -a "args" -e "environment"
     207  -a "args" An optional string of command line arguments to pass to
     208            the remote app.
     209  -d The (zero based) index of the Intel(R) Xeon Phi(TM) coprocessor to run the app on.
     210  -e "environment" An optional environment string to pass to the remote app.
     211      Multiple environment variable may be specified using spaces as separators:
     212        -e "LD_LIBRARY_PATH=/lib64/ DEBUG=1"
     213  -h Print this help message
     214  -l Do not execute the binary on the coprocessor. Instead, list the shared library
     215     dependency information.
     216  -p Disable console proxy.
     217  -t Time to wait for the remote app to finish (in seconds). After the timeout
     218     is reached the remote app will be terminated.
     219  -v Enable verbose mode. Note that verbose output will be displayed
     220     if the remote app terminates abnormally.
     221  -V Show version and build information
     222}}}
     223
     224
     225
     226
     227
    150228
    151229== Submitting an interactive job ==
     
    182260
    183261Note the prompt, "cypress01-100", in the above session. It is your interactive compute-node prompt. You can load modules, compile codes and test codes.
    184 `idev` transfers the environmental variables to computing node. Therefore, if you have loaded some module on the login node, you don't have to load the same module again.
     262`idev` transfers the environmental variables to computing node. Therefore, if you have loaded some modules on the login node, you don't have to load the same module again.
    185263
    186264==== Options ====
     
    213291}}}
    214292
     293==== MIC native run ====
     294
     295You can login to MIC device and run native codes. There are two MIC devices on each node, mic0 and mic1.
     296
     297{{{#!bash
     298[fuji@cypress01-100 nativeTest]$ ssh mic0
     299fuji@cypress01-100-mic0:~$
     300}}}
     301
     302The prompt, "cypress01-100-mic0" in the above session is your interactive MIC device prompt. Note that you cannot run CPU code there. Therefore, you cannot compile code on device, even compiling for native code.
     303The environmental variables are not set and also ''module'' command doesn't work. To run your native executable that uses shared libraries, you have to set environmental variables manually, like
     304{{{#!bash
     305export LD_LIBRARY_PATH=/share/apps/intel_parallel_studio_xe/2015_update1/lib/mic:$LD_LIBRARY_PATH
     306}}}
     307
    215308==== Questions or Suggestions ====
    216309