Changes between Initial Version and Version 1 of cypress/WorkingOnClusters


Ignore:
Timestamp:
Aug 19, 2015 9:37:30 AM (7 years ago)
Author:
cmaggio
Comment:

Legend:

Unmodified
Added
Removed
Modified
  • cypress/WorkingOnClusters

    v1 v1  
     1= Working on a *nix Cluster =
     2
     3By now we know how access a remote cluster. How to navigate a Linux environment. How to create and destroy files and directories. And how to transfer files from one machine to another. This constitutes the bulk of what you need to know to use a *nix cluster effectively, but not necessarily efficiently. There are a few more topics that can make your day to day life on the cluster much more pleasant.
     4
     5== Controlling the bash Environment ==
     6Recall that our default command line interpreter on Cypress is bash (the Bourne Again SHell).
     7
     8== Environment Variables ==
     9There are certain variables whose values determine the behavior of the bash environment. These are called ''environment variables'' and are traditionally denoted by ALL CAPS. To see the values of all our environment variables we can invoke the '''env''' command
     10
     11{{{
     12[tuhpc002@cypress1 ~]$ env
     13MODULE_VERSION_STACK=3.2.10
     14MANPATH=/cm/shared/apps/slurm/14.03.0/man:/usr/local/share/man:/usr/share/man/overrides:/usr/share/man/en:/usr/share/man:/cm/local/apps/environment-modules/current/share/man
     15HOSTNAME=cypress1
     16TERM=xterm-256color
     17SHELL=/bin/bash
     18HISTSIZE=1000
     19SSH_CLIENT=72.211.65.92 58652 22
     20LIBRARY_PATH=/cm/shared/apps/slurm/14.03.0/lib64/slurm:/cm/shared/apps/slurm/14.03.0/lib64
     21QTDIR=/usr/lib64/qt-3.3
     22QTINC=/usr/lib64/qt-3.3/include
     23SSH_TTY=/dev/pts/0
     24USER=tuhpc002
     25LD_LIBRARY_PATH=/cm/shared/apps/slurm/14.03.0/lib64/slurm:/cm/shared/apps/slurm/14.03.0/lib64
     26LS_COLORS=rs=0:di=38;5;27:ln=38;5;51:mh=44;38;5;15:pi=40;38;5;11:so=38;5;13:do=38;5;5:bd=48;5;232;38;5;11:cd=48;5;232;38;5;3:or=48;5;232;38;5;9:mi=05;48;5;232;38;5;15:su=48;5;196;38;5;15:sg=48;5;11;38;5;16:ca=48;5;196;38;5;226:tw=48;5;10;38;5;16:ow=48;5;10;38;5;21:st=48;5;21;38;5;15:ex=38;5;34:*.tar=38;5;9:*.tgz=38;5;9:*.arj=38;5;9:*.taz=38;5;9:*.lzh=38;5;9:*.lzma=38;5;9:*.tlz=38;5;9:*.txz=38;5;9:*.zip=38;5;9:*.z=38;5;9:*.Z=38;5;9:*.dz=38;5;9:*.gz=38;5;9:*.lz=38;5;9:*.xz=38;5;9:*.bz2=38;5;9:*.tbz=38;5;9:*.tbz2=38;5;9:*.bz=38;5;9:*.tz=38;5;9:*.deb=38;5;9:*.rpm=38;5;9:*.jar=38;5;9:*.rar=38;5;9:*.ace=38;5;9:*.zoo=38;5;9:*.cpio=38;5;9:*.7z=38;5;9:*.rz=38;5;9:*.jpg=38;5;13:*.jpeg=38;5;13:*.gif=38;5;13:*.bmp=38;5;13:*.pbm=38;5;13:*.pgm=38;5;13:*.ppm=38;5;13:*.tga=38;5;13:*.xbm=38;5;13:*.xpm=38;5;13:*.tif=38;5;13:*.tiff=38;5;13:*.png=38;5;13:*.svg=38;5;13:*.svgz=38;5;13:*.mng=38;5;13:*.pcx=38;5;13:*.mov=38;5;13:*.mpg=38;5;13:*.mpeg=38;5;13:*.m2v=38;5;13:*.mkv=38;5;13:*.ogm=38;5;13:*.mp4=38;5;13:*.m4v=38;5;13:*.mp4v=38;5;13:*.vob=38;5;13:*.qt=38;5;13:*.nuv=38;5;13:*.wmv=38;5;13:*.asf=38;5;13:*.rm=38;5;13:*.rmvb=38;5;13:*.flc=38;5;13:*.avi=38;5;13:*.fli=38;5;13:*.flv=38;5;13:*.gl=38;5;13:*.dl=38;5;13:*.xcf=38;5;13:*.xwd=38;5;13:*.yuv=38;5;13:*.cgm=38;5;13:*.emf=38;5;13:*.axv=38;5;13:*.anx=38;5;13:*.ogv=38;5;13:*.ogx=38;5;13:*.aac=38;5;45:*.au=38;5;45:*.flac=38;5;45:*.mid=38;5;45:*.midi=38;5;45:*.mka=38;5;45:*.mp3=38;5;45:*.mpc=38;5;45:*.ogg=38;5;45:*.ra=38;5;45:*.wav=38;5;45:*.axa=38;5;45:*.oga=38;5;45:*.spx=38;5;45:*.xspf=38;5;45:
     27CPATH=/cm/shared/apps/slurm/14.03.0/include
     28MODULE_VERSION=3.2.10
     29MAIL=/var/spool/mail/tuhpc002
     30PATH=/usr/lib64/qt-3.3/bin:/share/apps/bbcp/amd64_rhel60/bin:/share/apps/idev/bin:/cm/shared/apps/slurm/14.03.0/sbin:/cm/shared/apps/slurm/14.03.0/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/ibutils/bin:/opt/dell/srvadmin/bin:/home/workshop/tuhpc002/bin
     31PWD=/home/workshop/tuhpc002
     32_LMFILES_=/cm/shared/modulefiles/slurm/14.03.0:/share/apps/modulefiles/idev:/share/apps/modulefiles/bbcp/amd64_rhel60
     33LANG=en_US.UTF-8
     34MODULEPATH=/cm/local/modulefiles:/cm/shared/modulefiles:/share/apps/modulefiles
     35LOADEDMODULES=slurm/14.03.0:idev:bbcp/amd64_rhel60
     36SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass
     37HISTCONTROL=ignoredups
     38SHLVL=1
     39HOME=/home/workshop/tuhpc002
     40LOGNAME=tuhpc002
     41QTLIB=/usr/lib64/qt-3.3/lib
     42CVS_RSH=ssh
     43SSH_CONNECTION=72.211.65.92 58652 76.165.13.3 22
     44MODULESHOME=/cm/local/apps/environment-modules/3.2.10/Modules/3.2.10
     45LESSOPEN=|/usr/bin/lesspipe.sh %s
     46G_BROKEN_FILENAMES=1
     47module=() {  eval `/cm/local/apps/environment-modules/3.2.10/Modules/$MODULE_VERSION/bin/modulecmd bash $*`
     48}
     49_=/bin/env
     50OLDPWD=/home/workshop/tuhpc002/examples
     51[tuhpc002@cypress1 ~]$
     52}}}
     53
     54This list can easily be searched by "piping" ('''|''')it's output to '''grep'''
     55
     56{{{
     57[tuhpc002@cypress1 ~]$ env | grep PATH
     58MANPATH=/cm/shared/apps/slurm/14.03.0/man:/usr/local/share/man:/usr/share/man/overrides:/usr/share/man/en:/usr/share/man:/cm/local/apps/environment-modules/current/share/man
     59LIBRARY_PATH=/cm/shared/apps/slurm/14.03.0/lib64/slurm:/cm/shared/apps/slurm/14.03.0/lib64
     60LD_LIBRARY_PATH=/cm/shared/apps/slurm/14.03.0/lib64/slurm:/cm/shared/apps/slurm/14.03.0/lib64
     61CPATH=/cm/shared/apps/slurm/14.03.0/include
     62PATH=/usr/lib64/qt-3.3/bin:/share/apps/bbcp/amd64_rhel60/bin:/share/apps/idev/bin:/cm/shared/apps/slurm/14.03.0/sbin:/cm/shared/apps/slurm/14.03.0/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/ibutils/bin:/opt/dell/srvadmin/bin:/home/workshop/tuhpc002/bin
     63MODULEPATH=/cm/local/modulefiles:/cm/shared/modulefiles:/share/apps/modulefiles
     64[tuhpc002@cypress1 ~]$
     65}}}
     66
     67To use the information stored in the environment variable, we invoke it with the prefix $.
     68{{{
     69[tuhpc002@cypress1 examples]$ pwd
     70/home/workshop/tuhpc002/examples
     71[tuhpc002@cypress1 examples]$ cd $HOME
     72[tuhpc002@cypress1 ~]$ pwd
     73/home/workshop/tuhpc002
     74[tuhpc002@cypress1 ~]$
     75}}}
     76
     77The value of a single variable can be seen by using the '''echo''' command
     78{{{
     79[tuhpc002@cypress1 ~]$ echo $HOME
     80/home/workshop/tuhpc002
     81[tuhpc002@cypress1 ~]$
     82}}}
     83
     84An the value environment variable can be changed (or a new variable created) by invoking '''export'''.
     85
     86{{{
     87[tuhpc002@cypress1 ~]$ export EXAMPLEDIR=$HOME/examples/
     88[tuhpc002@cypress1 ~]$ echo $EXAMPLEDIR
     89/home/workshop/tuhpc002/examples/
     90[tuhpc002@cypress1 ~]$
     91}}}
     92
     93== The PATH variable ==
     94The PATH environment variable establishes a search path for executables. That is, when you type a command, the OS searches for that executable starting with the first directory listed in the PATH variable and moving along until it finds a match. Note that order matters, if multiple instances of an executable exist in the path, the one contained in the directory listed first will be used.
     95
     96{{{
     97[tuhpc002@cypress1 ~]$ echo $PATH
     98/usr/lib64/qt-3.3/bin:/share/apps/bbcp/amd64_rhel60/bin:/share/apps/idev/bin:/cm/shared/apps/slurm/14.03.0/sbin:/cm/shared/apps/slurm/14.03.0/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/ibutils/bin:/opt/dell/srvadmin/bin:/home/workshop/tuhpc002/bin
     99[tuhpc002@cypress1 ~]$
     100}}}
     101
     102
     103== bash_profile and bashrc ==
     104We can control the behavior of bash at login by editing the hidden configuration files in our home directory.
     105{{{
     106[tuhpc002@cypress1 ~]$ ls -al .bash*
     107-rw------- 1 tuhpc002 workshop 1259 Aug 18 22:34 .bash_history
     108-rw-r--r-- 1 tuhpc002 workshop   18 Jul 18  2013 .bash_logout
     109-rw-r--r-- 1 tuhpc002 workshop  176 Jul 18  2013 .bash_profile
     110-rw-r--r-- 1 tuhpc002 workshop  124 Sep 30  2014 .bashrc
     111[tuhpc002@cypress1 ~]$
     112}}}
     113
     114
     115The files we wish to concentrate on are ".bash_profile" and ".bashrc". Let's examine their contents
     116{{{
     117[tuhpc002@cypress1 ~]$ cat .bash_profile
     118# .bash_profile
     119
     120# Get the aliases and functions
     121if [ -f ~/.bashrc ]; then
     122        . ~/.bashrc
     123fi
     124
     125# User specific environment and startup programs
     126
     127PATH=$PATH:$HOME/bin
     128
     129export PATH
     130
     131[tuhpc002@cypress1 ~]$ cat .bashrc
     132# .bashrc
     133
     134# Source global definitions
     135if [ -f /etc/bashrc ]; then
     136        . /etc/bashrc
     137fi
     138
     139# User specific aliases and functions
     140[tuhpc002@cypress1 ~]$
     141}}}
     142
     143You can see that the two are nearly identical. The only difference being that our .bash_profile modifies the PATH environment variable.
     144
     145There is much confusion over the role that each of these files play. When you login to a system, the .bash_profile file is executed UPON LOGIN. The .bashrc file modifies the default environment when connected without login, for example when executing commands remotely using SSH. Note that changes made in the .bashrc file will be reflected in the .bash_profile file but not vise versa.
     146
     147== SSH config (Linux and Mac)==
     148It can be cumbersome to type your username and host address every time you login to a machine. The .ssh/config file offers the ability to create aliases to speed up this process.  Add the following to your .ssh/config file to create a shortcut
     149
     150{{{
     151  Host cypress1
     152  Hostname cypress1.tulane.edu
     153  User <your user ID on cypress>
     154}}}
     155
     156Now from your local machine you can simply type '''ssh cypres1''' and the username and hostname will be set automatically.