Changes between Initial Version and Version 1 of Workshops/IntroToHpc2015/Python


Ignore:
Timestamp:
Oct 12, 2015 3:59:22 PM (7 years ago)
Author:
pdejesus
Comment:

Legend:

Unmodified
Added
Removed
Modified
  • Workshops/IntroToHpc2015/Python

    v1 v1  
     1[[PageOutline]]
     2
     3= Running Python on Cypress =
     4
     5== Python Modules ==
     6As of August 18th, 2015 there are three (3) versions of Python available on Cypress
     7* Python 2.6.6 is loaded by default for al users
     8* Python 2.7.8 is available as part of the Anaconda module
     9* Python 2.7.10 is available as a stand alone module
     10
     11We currently do not have Python 3 installed on Cypress as we have had no requests for Python 3.
     12
     13=== Anaconda ===
     14As the name implies, Anaconda is a larger version of Python. In addition to Python 2.7.8, Anaconda includes over 300 of the most requested Python packages. This includes
     15* !NumPy
     16* Pandas
     17* !SciPy
     18* Matplotlib
     19* IPython.
     20
     21A complete list of packages available though Anaconda can be found [http://docs.continuum.io/anaconda/pkg-docs here].
     22
     23== Installing Packages ==
     24One of the best things about Python is the number of packages provided by the user community. On a personal machine, the most popular method today for managing these packages is the use of a package manager, like pip. Unfortunately, these require root access and are not a viable solution on Cypress.
     25
     26There is an easy solution. Any well written Python package will allow the user to perform a "user only installation." This typically only requires the addition of a single flag when installing the package.
     27
     28As an example, let's install the package Parallel Python (pp). If you wish to follow along, the package can be downloaded [http://www.parallelpython.com/content/view/18/32/ here].
     29
     30When you have the packages tar file in the desired directory, you can unpack it with the '''tar''' command
     31
     32{{{
     33[tulaneID@cypress1 ~]$ tar -xvf pp-1.6.4.tar.gz
     34pp-1.6.4/
     35pp-1.6.4/doc/
     36pp-1.6.4/doc/ppserver.1
     37pp-1.6.4/doc/example.config
     38pp-1.6.4/doc/ppdoc.html
     39pp-1.6.4/pp.py
     40pp-1.6.4/README
     41pp-1.6.4/pptransport.py
     42pp-1.6.4/PKG-INFO
     43pp-1.6.4/ppcommon.py
     44pp-1.6.4/ppserver.py
     45pp-1.6.4/ppworker.py
     46pp-1.6.4/CHANGELOG
     47pp-1.6.4/MANIFEST.in
     48pp-1.6.4/AUTHORS
     49pp-1.6.4/ppauto.py
     50pp-1.6.4/setup.py
     51pp-1.6.4/examples/
     52pp-1.6.4/examples/reverse_md5.py
     53pp-1.6.4/examples/sum_primes_functor.py
     54pp-1.6.4/examples/auto_diff.py
     55pp-1.6.4/examples/callback.py
     56pp-1.6.4/examples/quicksort.py
     57pp-1.6.4/examples/dynamic_ncpus.py
     58pp-1.6.4/examples/sum_primes.py
     59pp-1.6.4/COPYING
     60[tulaneID@cypress1 ~]$
     61}}}
     62
     63Now move into the package directory and examine the README file
     64
     65{{{
     66[tulaneID@cypress1 ~]$ cd pp-1.6.4
     67[tulaneID@cypress1 pp-1.6.4]$ ls
     68AUTHORS    COPYING  examples     PKG-INFO   ppcommon.py  ppserver.py     ppworker.py  setup.py
     69CHANGELOG  doc      MANIFEST.in  ppauto.py  pp.py        pptransport.py  README
     70[tulaneID@cypress1 pp-1.6.4]$ cat README
     71Visit http://www.parallelpython.com for up-to-date documentation, examples and support forums
     72
     73INSTALATION:
     74    python setup.py install
     75
     76LOCAL DOCUMENTATION:
     77    <htmlviewer> pydoc.html
     78}}}
     79
     80After using idev to launch and interactive session,let's try their instructions
     81
     82{{{
     83[tulaneID@cypress01-035 pp-1.6.4]$ python setup.py install
     84running install
     85running build
     86running build_py
     87creating build
     88creating build/lib
     89copying pp.py -> build/lib
     90copying ppauto.py -> build/lib
     91copying ppcommon.py -> build/lib
     92copying pptransport.py -> build/lib
     93copying ppworker.py -> build/lib
     94running build_scripts
     95creating build/scripts-2.7
     96copying and adjusting ppserver.py -> build/scripts-2.7
     97changing mode of build/scripts-2.7/ppserver.py from 644 to 755
     98running install_lib
     99copying build/lib/pptransport.py -> /share/apps/anaconda/2.1.0/lib/python2.7/site-packages
     100error: [Errno 13] Permission denied: '/share/apps/anaconda/2.1.0/lib/python2.7/site-packages/pptransport.py'
     101[tulaneID@cypress01-035 pp-1.6.4]$
     102}}}
     103
     104Our installation failed because we lack the permissions to add files to the system wide Python installation. But that's not what we want anyway, we just want to install the package for our personal use. We need to add the flag '''- -prefix=$HOME''' to tell the installer where to place the package.
     105
     106{{{
     107[tulaneID@cypress01-035 pp-1.6.4]$ python setup.py install --prefix=$HOME
     108running install
     109running build
     110running build_py
     111running build_scripts
     112running install_lib
     113creating /home/tulaneID/lib
     114creating /home/tulaneID/lib/python2.7
     115creating /home/tulaneID/lib/python2.7/site-packages
     116copying build/lib/pptransport.py -> /home/tulaneID/lib/python2.7/site-packages
     117copying build/lib/pp.py -> /home/tulaneID/lib/python2.7/site-packages
     118copying build/lib/ppworker.py -> /home/tulaneID/lib/python2.7/site-packages
     119copying build/lib/ppcommon.py -> /home/tulaneID/lib/python2.7/site-packages
     120copying build/lib/ppauto.py -> /home/tulaneID/lib/python2.7/site-packages
     121byte-compiling /home/tulaneID/lib/python2.7/site-packages/pptransport.py to pptransport.pyc
     122byte-compiling /home/tulaneID/lib/python2.7/site-packages/pp.py to pp.pyc
     123byte-compiling /home/tulaneID/lib/python2.7/site-packages/ppworker.py to ppworker.pyc
     124byte-compiling /home/tulaneID/lib/python2.7/site-packages/ppcommon.py to ppcommon.pyc
     125byte-compiling /home/tulaneID/lib/python2.7/site-packages/ppauto.py to ppauto.pyc
     126running install_scripts
     127copying build/scripts-2.7/ppserver.py -> /home/tulaneID/bin
     128changing mode of /home/tulaneID/bin/ppserver.py to 755
     129running install_egg_info
     130Writing /home/tulaneID/lib/python2.7/site-packages/pp-1.6.4-py2.7.egg-info
     131}}}
     132
     133We now have our own personal build of parallel python.
     134
     135{{{
     136[tulaneID@cypress01-035 pp-1.6.4]$ python
     137Python 2.7.8 |Anaconda 2.1.0 (64-bit)| (default, Aug 21 2014, 18:22:21)
     138[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2
     139Type "help", "copyright", "credits" or "license" for more information.
     140Anaconda is brought to you by Continuum Analytics.
     141Please check out: http://continuum.io/thanks and https://binstar.org
     142>>> import pp
     143>>> pp.version
     144'1.6.4'
     145>>>
     146
     147}}}
     148
     149== Running Python Interactively ==
     150
     151Start an interactive session using idev
     152
     153{{{
     154[tulaneID@cypress1 pp-1.6.4]$ idev
     155Requesting 1 node(s)  task(s) to normal queue of defq partition
     1561 task(s)/node, 20 cpu(s)/task, 2 MIC device(s)/node
     157Time: 0 (hr) 60 (min).
     158Submitted batch job 52311
     159Seems your requst is pending.
     160JOBID=52311 begin on cypress01-035
     161--> Creating interactive terminal session (login) on node cypress01-035.
     162--> You have 0 (hr) 60 (min).
     163[tulaneID@cypress01-035 pp-1.6.4]$
     164}}}
     165
     166Load the desired Python module
     167
     168{{{
     169[tulaneID@cypress01-035 pp-1.6.4]$ module load anaconda
     170[tulaneID@cypress01-035 pp-1.6.4]$ module list
     171Currently Loaded Modulefiles:
     172  1) git/2.4.1           3) idev                5) anaconda/2.1.0
     173  2) slurm/14.03.0       4) bbcp/amd64_rhel60
     174}}}
     175
     176Run Python in the command line window
     177
     178{{{
     179[tulaneID@cypress01-035 pp-1.6.4]$ python
     180Python 2.7.8 |Anaconda 2.1.0 (64-bit)| (default, Aug 21 2014, 18:22:21)
     181[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2
     182Type "help", "copyright", "credits" or "license" for more information.
     183Anaconda is brought to you by Continuum Analytics.
     184Please check out: http://continuum.io/thanks and https://binstar.org
     185>>>
     186}}}
     187
     188== Running a Python script in Batch mode ==
     189
     190You can also submit your Python job to the batch nodes (compute nodes) on Cypress. Inside your SLURM script, include a command to load the desired Python module. Then invoke '''python''' on your python script.
     191
     192{{{#!bash
     193#!/bin/bash
     194#SBATCH --qos=workshop          # Quality of Service
     195#SBATCH --partition=workshop    # Partition
     196#SBATCH --job-name=python       # Job Name
     197#SBATCH --time=00:01:00         # WallTime
     198#SBATCH --nodes=1               # Number of Nodes
     199#SBATCH --ntasks-per-node=1     # Number of tasks (MPI processes)
     200#SBATCH --cpus-per-task=1       # Number of threads per task (OMP threads)
     201
     202module load anaconda
     203python mypythonscript.py
     204}}}
     205
     206== Running a Parallel Python Job ==
     207
     208The exact configuration of you parallel job script will depend on the flavor of parallelism that you choose for you Python script.
     209
     210As an example we will use the Parallel Python (pp) package that we installed above. Parallel Python used the shared memory model of parallelism (analogous to to OpenMP). Let's run the [http://www.parallelpython.com/content/view/17/31/ sum of primes] example from the Parallel Python website.
     211
     212We need to communicate the number of cores we wish to use to our script. The syntax here is
     213
     214{{{
     215python sum_primes.py [ncpus]
     216}}}
     217
     218We can communicate the SLURM parameters to the script using the appropriate SLURM environment variable.
     219
     220{{{#!bash
     221#!/bin/bash
     222#SBATCH --qos=workshop          # Quality of Service
     223#SBATCH --partition=workshop    # Partition
     224#SBATCH --job-name=python       # Job Name
     225#SBATCH --time=00:01:00         # WallTime
     226#SBATCH --nodes=1               # Number of Nodes
     227#SBATCH --ntasks-per-node=1     # Number of tasks (MPI processes)
     228#SBATCH --cpus-per-task=15      # Number of threads per task (OMP threads)
     229
     230module load anaconda
     231python sum_primes.py $SLURM_CPUS_PER_TASK
     232}}}