Changes between Initial Version and Version 1 of cypress/WorkingOnClusters

Timestamp:

08/19/15 09:37:30 (9 years ago)

Author:

cmaggio

Comment:

—

cypress/WorkingOnClusters

@@ +1 @@
+= Working on a *nix Cluster =
+
+By now we know how access a remote cluster. How to navigate a Linux environment. How to create and destroy files and directories. And how to transfer files from one machine to another. This constitutes the bulk of what you need to know to use a *nix cluster effectively, but not necessarily efficiently. There are a few more topics that can make your day to day life on the cluster much more pleasant.
+
+== Controlling the bash Environment ==
+Recall that our default command line interpreter on Cypress is bash (the Bourne Again SHell).
+
+== Environment Variables ==
+There are certain variables whose values determine the behavior of the bash environment. These are called ''environment variables'' and are traditionally denoted by ALL CAPS. To see the values of all our environment variables we can invoke the '''env''' command
+
+{{{
+[tuhpc002@cypress1 ~]$ env
+MODULE_VERSION_STACK=3.2.10
+MANPATH=/cm/shared/apps/slurm/14.03.0/man:/usr/local/share/man:/usr/share/man/overrides:/usr/share/man/en:/usr/share/man:/cm/local/apps/environment-modules/current/share/man
+HOSTNAME=cypress1
+TERM=xterm-256color
+SHELL=/bin/bash
+HISTSIZE=1000
+SSH_CLIENT=72.211.65.92 58652 22
+LIBRARY_PATH=/cm/shared/apps/slurm/14.03.0/lib64/slurm:/cm/shared/apps/slurm/14.03.0/lib64
+QTDIR=/usr/lib64/qt-3.3
+QTINC=/usr/lib64/qt-3.3/include
+SSH_TTY=/dev/pts/0
+USER=tuhpc002
+LD_LIBRARY_PATH=/cm/shared/apps/slurm/14.03.0/lib64/slurm:/cm/shared/apps/slurm/14.03.0/lib64
+LS_COLORS=rs=0:di=38;5;27:ln=38;5;51:mh=44;38;5;15:pi=40;38;5;11:so=38;5;13:do=38;5;5:bd=48;5;232;38;5;11:cd=48;5;232;38;5;3:or=48;5;232;38;5;9:mi=05;48;5;232;38;5;15:su=48;5;196;38;5;15:sg=48;5;11;38;5;16:ca=48;5;196;38;5;226:tw=48;5;10;38;5;16:ow=48;5;10;38;5;21:st=48;5;21;38;5;15:ex=38;5;34:*.tar=38;5;9:*.tgz=38;5;9:*.arj=38;5;9:*.taz=38;5;9:*.lzh=38;5;9:*.lzma=38;5;9:*.tlz=38;5;9:*.txz=38;5;9:*.zip=38;5;9:*.z=38;5;9:*.Z=38;5;9:*.dz=38;5;9:*.gz=38;5;9:*.lz=38;5;9:*.xz=38;5;9:*.bz2=38;5;9:*.tbz=38;5;9:*.tbz2=38;5;9:*.bz=38;5;9:*.tz=38;5;9:*.deb=38;5;9:*.rpm=38;5;9:*.jar=38;5;9:*.rar=38;5;9:*.ace=38;5;9:*.zoo=38;5;9:*.cpio=38;5;9:*.7z=38;5;9:*.rz=38;5;9:*.jpg=38;5;13:*.jpeg=38;5;13:*.gif=38;5;13:*.bmp=38;5;13:*.pbm=38;5;13:*.pgm=38;5;13:*.ppm=38;5;13:*.tga=38;5;13:*.xbm=38;5;13:*.xpm=38;5;13:*.tif=38;5;13:*.tiff=38;5;13:*.png=38;5;13:*.svg=38;5;13:*.svgz=38;5;13:*.mng=38;5;13:*.pcx=38;5;13:*.mov=38;5;13:*.mpg=38;5;13:*.mpeg=38;5;13:*.m2v=38;5;13:*.mkv=38;5;13:*.ogm=38;5;13:*.mp4=38;5;13:*.m4v=38;5;13:*.mp4v=38;5;13:*.vob=38;5;13:*.qt=38;5;13:*.nuv=38;5;13:*.wmv=38;5;13:*.asf=38;5;13:*.rm=38;5;13:*.rmvb=38;5;13:*.flc=38;5;13:*.avi=38;5;13:*.fli=38;5;13:*.flv=38;5;13:*.gl=38;5;13:*.dl=38;5;13:*.xcf=38;5;13:*.xwd=38;5;13:*.yuv=38;5;13:*.cgm=38;5;13:*.emf=38;5;13:*.axv=38;5;13:*.anx=38;5;13:*.ogv=38;5;13:*.ogx=38;5;13:*.aac=38;5;45:*.au=38;5;45:*.flac=38;5;45:*.mid=38;5;45:*.midi=38;5;45:*.mka=38;5;45:*.mp3=38;5;45:*.mpc=38;5;45:*.ogg=38;5;45:*.ra=38;5;45:*.wav=38;5;45:*.axa=38;5;45:*.oga=38;5;45:*.spx=38;5;45:*.xspf=38;5;45:
+CPATH=/cm/shared/apps/slurm/14.03.0/include
+MODULE_VERSION=3.2.10
+MAIL=/var/spool/mail/tuhpc002
+PATH=/usr/lib64/qt-3.3/bin:/share/apps/bbcp/amd64_rhel60/bin:/share/apps/idev/bin:/cm/shared/apps/slurm/14.03.0/sbin:/cm/shared/apps/slurm/14.03.0/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/ibutils/bin:/opt/dell/srvadmin/bin:/home/workshop/tuhpc002/bin
+PWD=/home/workshop/tuhpc002
+_LMFILES_=/cm/shared/modulefiles/slurm/14.03.0:/share/apps/modulefiles/idev:/share/apps/modulefiles/bbcp/amd64_rhel60
+LANG=en_US.UTF-8
+MODULEPATH=/cm/local/modulefiles:/cm/shared/modulefiles:/share/apps/modulefiles
+LOADEDMODULES=slurm/14.03.0:idev:bbcp/amd64_rhel60
+SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass
+HISTCONTROL=ignoredups
+SHLVL=1
+HOME=/home/workshop/tuhpc002
+LOGNAME=tuhpc002
+QTLIB=/usr/lib64/qt-3.3/lib
+CVS_RSH=ssh
+SSH_CONNECTION=72.211.65.92 58652 76.165.13.3 22
+MODULESHOME=/cm/local/apps/environment-modules/3.2.10/Modules/3.2.10
+LESSOPEN=|/usr/bin/lesspipe.sh %s
+G_BROKEN_FILENAMES=1
+module=() {  eval `/cm/local/apps/environment-modules/3.2.10/Modules/$MODULE_VERSION/bin/modulecmd bash $*`
+}
+_=/bin/env
+OLDPWD=/home/workshop/tuhpc002/examples
+[tuhpc002@cypress1 ~]$ 
+}}}
+
+This list can easily be searched by "piping" ('''|''')it's output to '''grep'''
+
+{{{
+[tuhpc002@cypress1 ~]$ env | grep PATH
+MANPATH=/cm/shared/apps/slurm/14.03.0/man:/usr/local/share/man:/usr/share/man/overrides:/usr/share/man/en:/usr/share/man:/cm/local/apps/environment-modules/current/share/man
+LIBRARY_PATH=/cm/shared/apps/slurm/14.03.0/lib64/slurm:/cm/shared/apps/slurm/14.03.0/lib64
+LD_LIBRARY_PATH=/cm/shared/apps/slurm/14.03.0/lib64/slurm:/cm/shared/apps/slurm/14.03.0/lib64
+CPATH=/cm/shared/apps/slurm/14.03.0/include
+PATH=/usr/lib64/qt-3.3/bin:/share/apps/bbcp/amd64_rhel60/bin:/share/apps/idev/bin:/cm/shared/apps/slurm/14.03.0/sbin:/cm/shared/apps/slurm/14.03.0/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/ibutils/bin:/opt/dell/srvadmin/bin:/home/workshop/tuhpc002/bin
+MODULEPATH=/cm/local/modulefiles:/cm/shared/modulefiles:/share/apps/modulefiles
+[tuhpc002@cypress1 ~]$ 
+}}}
+
+To use the information stored in the environment variable, we invoke it with the prefix $.
+{{{
+[tuhpc002@cypress1 examples]$ pwd
+/home/workshop/tuhpc002/examples
+[tuhpc002@cypress1 examples]$ cd $HOME
+[tuhpc002@cypress1 ~]$ pwd
+/home/workshop/tuhpc002
+[tuhpc002@cypress1 ~]$ 
+}}}
+
+The value of a single variable can be seen by using the '''echo''' command
+{{{
+[tuhpc002@cypress1 ~]$ echo $HOME
+/home/workshop/tuhpc002
+[tuhpc002@cypress1 ~]$ 
+}}}
+
+An the value environment variable can be changed (or a new variable created) by invoking '''export'''.
+
+{{{
+[tuhpc002@cypress1 ~]$ export EXAMPLEDIR=$HOME/examples/
+[tuhpc002@cypress1 ~]$ echo $EXAMPLEDIR
+/home/workshop/tuhpc002/examples/
+[tuhpc002@cypress1 ~]$ 
+}}}
+
+== The PATH variable ==
+The PATH environment variable establishes a search path for executables. That is, when you type a command, the OS searches for that executable starting with the first directory listed in the PATH variable and moving along until it finds a match. Note that order matters, if multiple instances of an executable exist in the path, the one contained in the directory listed first will be used.
+
+{{{
+[tuhpc002@cypress1 ~]$ echo $PATH
+/usr/lib64/qt-3.3/bin:/share/apps/bbcp/amd64_rhel60/bin:/share/apps/idev/bin:/cm/shared/apps/slurm/14.03.0/sbin:/cm/shared/apps/slurm/14.03.0/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/ibutils/bin:/opt/dell/srvadmin/bin:/home/workshop/tuhpc002/bin
+[tuhpc002@cypress1 ~]$ 
+}}}
+
+
+== bash_profile and bashrc ==
+We can control the behavior of bash at login by editing the hidden configuration files in our home directory.
+{{{
+[tuhpc002@cypress1 ~]$ ls -al .bash*
+-rw------- 1 tuhpc002 workshop 1259 Aug 18 22:34 .bash_history
+-rw-r--r-- 1 tuhpc002 workshop   18 Jul 18  2013 .bash_logout
+-rw-r--r-- 1 tuhpc002 workshop  176 Jul 18  2013 .bash_profile
+-rw-r--r-- 1 tuhpc002 workshop  124 Sep 30  2014 .bashrc
+[tuhpc002@cypress1 ~]$ 
+}}}
+
+
+The files we wish to concentrate on are ".bash_profile" and ".bashrc". Let's examine their contents
+{{{
+[tuhpc002@cypress1 ~]$ cat .bash_profile 
+# .bash_profile
+
+# Get the aliases and functions
+if [ -f ~/.bashrc ]; then
+        . ~/.bashrc
+fi
+
+# User specific environment and startup programs
+
+PATH=$PATH:$HOME/bin
+
+export PATH
+
+[tuhpc002@cypress1 ~]$ cat .bashrc 
+# .bashrc
+
+# Source global definitions
+if [ -f /etc/bashrc ]; then
+        . /etc/bashrc
+fi
+
+# User specific aliases and functions
+[tuhpc002@cypress1 ~]$
+}}}
+
+You can see that the two are nearly identical. The only difference being that our .bash_profile modifies the PATH environment variable. 
+
+There is much confusion over the role that each of these files play. When you login to a system, the .bash_profile file is executed UPON LOGIN. The .bashrc file modifies the default environment when connected without login, for example when executing commands remotely using SSH. Note that changes made in the .bashrc file will be reflected in the .bash_profile file but not vise versa. 
+
+== SSH config (Linux and Mac)==
+It can be cumbersome to type your username and host address every time you login to a machine. The .ssh/config file offers the ability to create aliases to speed up this process.  Add the following to your .ssh/config file to create a shortcut
+
+{{{
+  Host cypress1
+  Hostname cypress1.tulane.edu
+  User <your user ID on cypress>
+}}}
+
+Now from your local machine you can simply type '''ssh cypres1''' and the username and hostname will be set automatically.