Changes between Version 12 and Version 13 of cypress/XeonPhi

Timestamp:

08/21/15 08:22:13 (9 years ago)

Author:

cmaggio

Comment:

—

cypress/XeonPhi

@@ -1 +1 @@
 = Programming for the Xeon Phi Coprocessor on Cypress =
+
+==== Workshop Reminder ====
+To take advantage of the workshop QOS:
+{{{#!bash
+export MY_PARTITION=workshop
+export MY_QUEUE=workshop
+idev -c 4 --gres=mic:0
+}}}
+
 The Xeon Phi coprocessor is an accelerator used to provide many cores to parallel applications. While it has fewer threads available than a typical GPU accelerator, the processors are much "smarter" and the programming paradigm is similar to what you experience coding for CPUs.
 
@@ -30 +39 @@
 * Knights Hill (next gen)
 
-You'll typically hear us call it either the MIC or the Phi.
+You'll typically hear us call the 7120p either the ''MIC'' or the ''Phi''. This is to help distinguish it from the Xeon E5 processors which we'll refer to as the ''host''.
+
 == Xeon Phi Usage Models ==
 The intel suite provides parallel instantiations and compilers that support three distinct programming models:
@@ -39 +49 @@
 
 === Automatic Offloading ===
+==== Eligibility ====
+As we saw yesterday during our Matlab tutorial, any program/code that makes use of the Intel MKL library may take advantage of Automatic Offloading (AO) to the MIC. However, not every MKL routine will automatically offload. The Routines that are eligible for AO are:
+* BLAS:
+ * BLAS level-3 subroutines - ?SYMM,?TRMM, ?TRSM, ?GEMM
+* LAPACK:
+ * LU (?GETRF), Cholesky ((S/D)POTRF), and QR (?GEQRF) factorization functions
+
+However, AO will only kick in if MKL deems the problem to be of sufficient size (i.e. the increase in parallelism will outweigh the increase in overhead). For instance, SGEMM will use AO only if the matrix size exceeds 2048x2048. For more information on which routines are eligible for AO see the white paper [https://software.intel.com/en-us/articles/intel-mkl-automatic-offload-enabled-functions-for-intel-xeon-phi-coprocessors|Intel® MKL Automatic Offload enabled functions for Intel Xeon Phi coprocessors] 
+
+==== Enabling Offloading ====
+To enable AO on Cypress you must
+* Load the Intel Parallel Studio XE module
+* Turn on MKL AO by setting the environment variable MKL_MIC_ENABLE to 1 (0 or nothing will turn off MKL AO)
+* (OPTIONAL) Turn on offload reporting to track your use of the MIC by setting OFFLOAD_REPORT to either 1 or 2. Setting OFFLOAD_REPORT to 2 adds more detail than 1 and will give you information on data transfers.
+{{{
+[tulaneID@cypress1]$ module load intel-psxe
+[tulaneID@cypress1]$ export MKL_MIC_ENABLE=1
+[tulaneID@cypress1]$ export OFFLOAD_REPORT=2
+}}} 
+
+==== Example using SGEMM ====
+Let's do a small example using SGEMM to test the behavior of MLK AO
+
+{{{#!c
+[tuhpc002@cypress01-089 Day2]$ cat sgemm_example.c 
+/* System headers */
+#include <stdio.h>
+#include <stdlib.h>
+#include <malloc.h>
+#include <stdint.h>
+
+#include "mkl.h"
+
+// dtime
+//
+// returns the current wall clock time
+//
+double dtime()
+{
+    double tseconds = 0.0;
+    struct timeval mytime;
+    gettimeofday(&mytime,(struct timezone*)0);
+    tseconds = (double)(mytime.tv_sec +
+                mytime.tv_usec*1.0e-6);
+    return( tseconds );
+}
+
+int main(int argc, char **argv)
+{
+        float *A, *B, *C; /* Matrices */
+        double workdivision;
+        double tstart, tstop, ttime;
+
+        MKL_INT N = 2560; /* Matrix dimensions */
+        MKL_INT LD = N; /* Leading dimension */
+        int matrix_bytes; /* Matrix size in bytes */
+        int matrix_elements; /* Matrix size in elements */
+
+        float alpha = 1.0, beta = 1.0; /* Scaling factors */
+        char transa = 'N', transb = 'N'; /* Transposition options */
+
+        int i, j; /* Counters */
+
+        matrix_elements = N * N;
+        matrix_bytes = sizeof(float) * matrix_elements;
+
+        /* Allocate the matrices */
+        A = malloc(matrix_bytes);
+        B = malloc(matrix_bytes);
+        C = malloc(matrix_bytes);
+
+        /* Initialize the matrices */
+        for (i = 0; i < matrix_elements; i++) {
+                A[i] = 1.0; B[i] = 2.0; C[i] = 0.0;
+        }
+        
+        tstart = dtime();
+        sgemm(&transa, &transb, &N, &N, &N, &alpha, A, &N, B, &N,
+                        &beta, C, &N);
+        tstop = dtime();
+                /* Free the matrix memory */
+        free(A); free(B); free(C);
+
+        // elasped time
+        ttime = tstop - tstart;
+        //
+        // Print the results
+        //
+        if ((ttime) > 0.0)
+        {
+                printf("Time spent on SGEMM = %10.3lf\n",ttime);
+        }
+        printf("Done\n");
+        
+    return 0;
+}
+}}}
+
+To test MKL AO 
+* Get onto a compute node using idev
+{{{
+[tuhpc002@cypress1 Day2]$ export MY_PARTITION=workshop
+[tuhpc002@cypress1 Day2]$ export MY_QUEUE=workshop
+[tuhpc002@cypress1 Day2]$ idev -c 4 --gres=mic:0
+Requesting 1 node(s)  task(s) to workshop queue of workshop partition
+1 task(s)/node, 4 cpu(s)/task, 2 MIC device(s)/node
+Time: 0 (hr) 60 (min).
+Submitted batch job 54982
+JOBID=54982 begin on cypress01-089
+--> Creating interactive terminal session (login) on node cypress01-089.
+--> You have 0 (hr) 60 (min).
+Last login: Fri Aug 21 07:16:58 2015 from cypress1.cm.cluster
+[tuhpc002@cypress01-089 Day2]$ 
+}}}
+
+Note: We will be sharing MICs so expect some resource conflicts
+
+* Load the Intel module containing MKL and set your environment variables
+{{{
+[tuhpc002@cypress01-089 Day2]$ module load intel-psxe
+[tuhpc002@cypress01-089 Day2]$ export MKL_MIC_ENABLE=0
+[tuhpc002@cypress01-089 Day2]$ export OFFLOAD_REPORT=2
+}}}
+
+Notice that automatic offloading is turned OFF. This will set our baseline.
+* Compile the example code being sure to link to the MKL library
+* Run the executable
+* Turn on MKL AO and run it again
+{{{
+[tuhpc002@cypress01-089 Day2]$ icc -O3 -mkl -openmp sgemm_example.c -o AOtest
+[tuhpc002@cypress01-089 Day2]$ ./AOtest 
+Time spent on SGEMM =      0.835
+Done
+[tuhpc002@cypress01-089 Day2]$ export MKL_MIC_ENABLE=1
+[tuhpc002@cypress01-089 Day2]$ ./AOtest 
+[MKL] [MIC --] [AO Function]    SGEMM
+[MKL] [MIC --] [AO SGEMM Workdivision]  0.60 0.20 0.20
+[MKL] [MIC 00] [AO SGEMM CPU Time]      2.858848 seconds
+[MKL] [MIC 00] [AO SGEMM MIC Time]      0.104307 seconds
+[MKL] [MIC 00] [AO SGEMM CPU->MIC Data] 31457280 bytes
+[MKL] [MIC 00] [AO SGEMM MIC->CPU Data] 5242880 bytes
+[MKL] [MIC 01] [AO SGEMM CPU Time]      2.858848 seconds
+[MKL] [MIC 01] [AO SGEMM MIC Time]      0.113478 seconds
+[MKL] [MIC 01] [AO SGEMM CPU->MIC Data] 31457280 bytes
+[MKL] [MIC 01] [AO SGEMM MIC->CPU Data] 5242880 bytes
+Time spent on SGEMM =      3.436
+Done
+[tuhpc002@cypress01-089 Day2]$ 
+}}}
+
+The Point: This example gets at some of the challenges of coding for the Xeon Phi. Utilization is simple, but optimization can be a real challenge. Let's look at a few more options we can manipulate through environment variables:
+
+* The work division among the Host and MICs can also be tuned by hand using MKL_MIC_<0,1>_WORKDIVISION
+{{{
+[tuhpc002@cypress01-089 Day2]$ export MKL_MIC_0_WORKDIVISION=1.0
+[tuhpc002@cypress01-089 Day2]$ ./AOtest 
+[MKL] [MIC --] [AO Function]    SGEMM
+[MKL] [MIC --] [AO SGEMM Workdivision]  0.00 1.00 0.00
+[MKL] [MIC 00] [AO SGEMM CPU Time]      2.831957 seconds
+[MKL] [MIC 00] [AO SGEMM MIC Time]      0.141694 seconds
+[MKL] [MIC 00] [AO SGEMM CPU->MIC Data] 52428800 bytes
+[MKL] [MIC 00] [AO SGEMM MIC->CPU Data] 26214400 bytes
+[MKL] [MIC 01] [AO SGEMM CPU Time]      2.831957 seconds
+[MKL] [MIC 01] [AO SGEMM MIC Time]      0.000000 seconds
+[MKL] [MIC 01] [AO SGEMM CPU->MIC Data] 0 bytes
+[MKL] [MIC 01] [AO SGEMM MIC->CPU Data] 0 bytes
+Time spent on SGEMM =      3.394
+}}}
+
+* The number of threads used on each MIC can be controlled using MIC_OMP_NUMTHREADS
+{{{
+[tuhpc002@cypress01-089 Day2]$ export MIC_OMP_NUMTHREADS=122
+[tuhpc002@cypress01-089 Day2]$ ./AOtest 
+[MKL] [MIC --] [AO Function]    SGEMM
+[MKL] [MIC --] [AO SGEMM Workdivision]  0.60 0.20 0.20
+[MKL] [MIC 00] [AO SGEMM CPU Time]      1.625511 seconds
+[MKL] [MIC 00] [AO SGEMM MIC Time]      0.102266 seconds
+[MKL] [MIC 00] [AO SGEMM CPU->MIC Data] 31457280 bytes
+[MKL] [MIC 00] [AO SGEMM MIC->CPU Data] 5242880 bytes
+[MKL] [MIC 01] [AO SGEMM CPU Time]      1.625511 seconds
+[MKL] [MIC 01] [AO SGEMM MIC Time]      0.089364 seconds
+[MKL] [MIC 01] [AO SGEMM CPU->MIC Data] 31457280 bytes
+[MKL] [MIC 01] [AO SGEMM MIC->CPU Data] 5242880 bytes
+Time spent on SGEMM =      2.288
+Done
+[tuhpc002@cypress01-089 Day2]$
+}}}
+
+* We can control the distribution of threads using MIC_KMP_AFFINITY
+{{{
+[tuhpc002@cypress01-089 Day2]$ export MIC_KMP_AFFINITY=scatter
+[tuhpc002@cypress01-089 Day2]$ ./AOtest 
+[MKL] [MIC --] [AO Function]    SGEMM
+[MKL] [MIC --] [AO SGEMM Workdivision]  0.60 0.20 0.20
+[MKL] [MIC 00] [AO SGEMM CPU Time]      1.631954 seconds
+[MKL] [MIC 00] [AO SGEMM MIC Time]      0.101270 seconds
+[MKL] [MIC 00] [AO SGEMM CPU->MIC Data] 31457280 bytes
+[MKL] [MIC 00] [AO SGEMM MIC->CPU Data] 5242880 bytes
+[MKL] [MIC 01] [AO SGEMM CPU Time]      1.631954 seconds
+[MKL] [MIC 01] [AO SGEMM MIC Time]      0.105702 seconds
+[MKL] [MIC 01] [AO SGEMM CPU->MIC Data] 31457280 bytes
+[MKL] [MIC 01] [AO SGEMM MIC->CPU Data] 5242880 bytes
+Time spent on SGEMM =      2.028
+Done
+[tuhpc002@cypress01-089 Day2]$
+}}}
 
 === Native Programming ===
+
+The native model centers around the notion that each MIC is its own machine with it's own architecture. The first challenge is to compile code to run specifically on the hardware of the MIC.
+
+
+
+SLURM jobscript script is, for example,
+
+{{{#!bash
+#!/bin/bash
+#SBATCH --qos=normal            # Quality of Service
+#SBATCH --job-name=nativeTest   # Job Name
+#SBATCH --time=00:10:00         # WallTime
+#SBATCH --nodes=1               # Number of Nodes
+#SBATCH --ntasks-per-node=1     # Number of tasks (MPI presseces)
+#SBATCH --cpus-per-task=1       # Number of processors per task OpenMP threads()
+#SBATCH --gres=mic:1            # Number of Co-Processors
+
+micnativeloadex ./myNativeExecutable -e "OMP_NUM_THREADS=100" -d 0 -v
+}}}
+
+
+
+In the script above we request one MIC device that will be device number 0. 
+"micnativeloadex" command launches MIC native executable. "-e "OMP_NUM_THREADS=100"" option to set the number of threads on the MIC device to 100.
+For more options, see below.
+
+{{{#!bash
+[fuji@cypress01-090 nativeTest]$ micnativeloadex -h
+
+Usage:
+micnativeloadex [ -h | -V ] AppName -l -t timeout -p -v -d coprocessor -a "args" -e "environment"
+  -a "args" An optional string of command line arguments to pass to
+            the remote app.
+  -d The (zero based) index of the Intel(R) Xeon Phi(TM) coprocessor to run the app on.
+  -e "environment" An optional environment string to pass to the remote app.
+      Multiple environment variable may be specified using spaces as separators:
+        -e "LD_LIBRARY_PATH=/lib64/ DEBUG=1"
+  -h Print this help message
+  -l Do not execute the binary on the coprocessor. Instead, list the shared library
+     dependency information.
+  -p Disable console proxy.
+  -t Time to wait for the remote app to finish (in seconds). After the timeout
+     is reached the remote app will be terminated.
+  -v Enable verbose mode. Note that verbose output will be displayed
+     if the remote app terminates abnormally.
+  -V Show version and build information
+}}}
+
+
+
 
 === Offloading ===