= Working on a *nix Cluster = By now we know how access a remote cluster. How to navigate a Linux environment. How to create and destroy files and directories. And how to transfer files from one machine to another. This constitutes the bulk of what you need to know to use a *nix cluster effectively, but not necessarily efficiently. There are a few more topics that can make your day to day life on the cluster much more pleasant. == Controlling the bash Environment == Recall that our default command line interpreter on Cypress is bash (the Bourne Again SHell). == Environment Variables == There are certain variables whose values determine the behavior of the bash environment. These are called ''environment variables'' and are traditionally denoted by ALL CAPS. To see the values of all our environment variables we can invoke the '''env''' command {{{ [tuhpc002@cypress1 ~]$ env MODULE_VERSION_STACK=3.2.10 MANPATH=/cm/shared/apps/slurm/14.03.0/man:/usr/local/share/man:/usr/share/man/overrides:/usr/share/man/en:/usr/share/man:/cm/local/apps/environment-modules/current/share/man HOSTNAME=cypress1 TERM=xterm-256color SHELL=/bin/bash HISTSIZE=1000 SSH_CLIENT=72.211.65.92 58652 22 LIBRARY_PATH=/cm/shared/apps/slurm/14.03.0/lib64/slurm:/cm/shared/apps/slurm/14.03.0/lib64 QTDIR=/usr/lib64/qt-3.3 QTINC=/usr/lib64/qt-3.3/include SSH_TTY=/dev/pts/0 USER=tuhpc002 LD_LIBRARY_PATH=/cm/shared/apps/slurm/14.03.0/lib64/slurm:/cm/shared/apps/slurm/14.03.0/lib64 LS_COLORS=rs=0:di=38;5;27:ln=38;5;51:mh=44;38;5;15:pi=40;38;5;11:so=38;5;13:do=38;5;5:bd=48;5;232;38;5;11:cd=48;5;232;38;5;3:or=48;5;232;38;5;9:mi=05;48;5;232;38;5;15:su=48;5;196;38;5;15:sg=48;5;11;38;5;16:ca=48;5;196;38;5;226:tw=48;5;10;38;5;16:ow=48;5;10;38;5;21:st=48;5;21;38;5;15:ex=38;5;34:*.tar=38;5;9:*.tgz=38;5;9:*.arj=38;5;9:*.taz=38;5;9:*.lzh=38;5;9:*.lzma=38;5;9:*.tlz=38;5;9:*.txz=38;5;9:*.zip=38;5;9:*.z=38;5;9:*.Z=38;5;9:*.dz=38;5;9:*.gz=38;5;9:*.lz=38;5;9:*.xz=38;5;9:*.bz2=38;5;9:*.tbz=38;5;9:*.tbz2=38;5;9:*.bz=38;5;9:*.tz=38;5;9:*.deb=38;5;9:*.rpm=38;5;9:*.jar=38;5;9:*.rar=38;5;9:*.ace=38;5;9:*.zoo=38;5;9:*.cpio=38;5;9:*.7z=38;5;9:*.rz=38;5;9:*.jpg=38;5;13:*.jpeg=38;5;13:*.gif=38;5;13:*.bmp=38;5;13:*.pbm=38;5;13:*.pgm=38;5;13:*.ppm=38;5;13:*.tga=38;5;13:*.xbm=38;5;13:*.xpm=38;5;13:*.tif=38;5;13:*.tiff=38;5;13:*.png=38;5;13:*.svg=38;5;13:*.svgz=38;5;13:*.mng=38;5;13:*.pcx=38;5;13:*.mov=38;5;13:*.mpg=38;5;13:*.mpeg=38;5;13:*.m2v=38;5;13:*.mkv=38;5;13:*.ogm=38;5;13:*.mp4=38;5;13:*.m4v=38;5;13:*.mp4v=38;5;13:*.vob=38;5;13:*.qt=38;5;13:*.nuv=38;5;13:*.wmv=38;5;13:*.asf=38;5;13:*.rm=38;5;13:*.rmvb=38;5;13:*.flc=38;5;13:*.avi=38;5;13:*.fli=38;5;13:*.flv=38;5;13:*.gl=38;5;13:*.dl=38;5;13:*.xcf=38;5;13:*.xwd=38;5;13:*.yuv=38;5;13:*.cgm=38;5;13:*.emf=38;5;13:*.axv=38;5;13:*.anx=38;5;13:*.ogv=38;5;13:*.ogx=38;5;13:*.aac=38;5;45:*.au=38;5;45:*.flac=38;5;45:*.mid=38;5;45:*.midi=38;5;45:*.mka=38;5;45:*.mp3=38;5;45:*.mpc=38;5;45:*.ogg=38;5;45:*.ra=38;5;45:*.wav=38;5;45:*.axa=38;5;45:*.oga=38;5;45:*.spx=38;5;45:*.xspf=38;5;45: CPATH=/cm/shared/apps/slurm/14.03.0/include MODULE_VERSION=3.2.10 MAIL=/var/spool/mail/tuhpc002 PATH=/usr/lib64/qt-3.3/bin:/share/apps/bbcp/amd64_rhel60/bin:/share/apps/idev/bin:/cm/shared/apps/slurm/14.03.0/sbin:/cm/shared/apps/slurm/14.03.0/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/ibutils/bin:/opt/dell/srvadmin/bin:/home/workshop/tuhpc002/bin PWD=/home/workshop/tuhpc002 _LMFILES_=/cm/shared/modulefiles/slurm/14.03.0:/share/apps/modulefiles/idev:/share/apps/modulefiles/bbcp/amd64_rhel60 LANG=en_US.UTF-8 MODULEPATH=/cm/local/modulefiles:/cm/shared/modulefiles:/share/apps/modulefiles LOADEDMODULES=slurm/14.03.0:idev:bbcp/amd64_rhel60 SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass HISTCONTROL=ignoredups SHLVL=1 HOME=/home/workshop/tuhpc002 LOGNAME=tuhpc002 QTLIB=/usr/lib64/qt-3.3/lib CVS_RSH=ssh SSH_CONNECTION=72.211.65.92 58652 76.165.13.3 22 MODULESHOME=/cm/local/apps/environment-modules/3.2.10/Modules/3.2.10 LESSOPEN=|/usr/bin/lesspipe.sh %s G_BROKEN_FILENAMES=1 module=() { eval `/cm/local/apps/environment-modules/3.2.10/Modules/$MODULE_VERSION/bin/modulecmd bash $*` } _=/bin/env OLDPWD=/home/workshop/tuhpc002/examples [tuhpc002@cypress1 ~]$ }}} This list can easily be searched by "piping" ('''|''')it's output to '''grep''' {{{ [tuhpc002@cypress1 ~]$ env | grep PATH MANPATH=/cm/shared/apps/slurm/14.03.0/man:/usr/local/share/man:/usr/share/man/overrides:/usr/share/man/en:/usr/share/man:/cm/local/apps/environment-modules/current/share/man LIBRARY_PATH=/cm/shared/apps/slurm/14.03.0/lib64/slurm:/cm/shared/apps/slurm/14.03.0/lib64 LD_LIBRARY_PATH=/cm/shared/apps/slurm/14.03.0/lib64/slurm:/cm/shared/apps/slurm/14.03.0/lib64 CPATH=/cm/shared/apps/slurm/14.03.0/include PATH=/usr/lib64/qt-3.3/bin:/share/apps/bbcp/amd64_rhel60/bin:/share/apps/idev/bin:/cm/shared/apps/slurm/14.03.0/sbin:/cm/shared/apps/slurm/14.03.0/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/ibutils/bin:/opt/dell/srvadmin/bin:/home/workshop/tuhpc002/bin MODULEPATH=/cm/local/modulefiles:/cm/shared/modulefiles:/share/apps/modulefiles [tuhpc002@cypress1 ~]$ }}} To use the information stored in the environment variable, we invoke it with the prefix $. {{{ [tuhpc002@cypress1 examples]$ pwd /home/workshop/tuhpc002/examples [tuhpc002@cypress1 examples]$ cd $HOME [tuhpc002@cypress1 ~]$ pwd /home/workshop/tuhpc002 [tuhpc002@cypress1 ~]$ }}} The value of a single variable can be seen by using the '''echo''' command {{{ [tuhpc002@cypress1 ~]$ echo $HOME /home/workshop/tuhpc002 [tuhpc002@cypress1 ~]$ }}} Also the value of an environment variable can be changed (or a new variable created) by invoking '''export'''. In general such changes will remain in effect throughout the remainder of your session - as well as for the commands and programs you execute therein. {{{ [tuhpc002@cypress1 ~]$ export EXAMPLEDIR=$HOME/examples/ [tuhpc002@cypress1 ~]$ echo $EXAMPLEDIR /home/workshop/tuhpc002/examples/ [tuhpc002@cypress1 ~]$ }}} == The PATH variable == The PATH environment variable establishes a search path for executables. That is, when you type a command, the OS searches for that executable starting with the first directory listed in the PATH variable and moving along until it finds a match. Note that order matters, if multiple instances of an executable exist in the path, the one contained in the directory listed first will be used. {{{ [tuhpc002@cypress1 ~]$ echo $PATH /usr/lib64/qt-3.3/bin:/share/apps/bbcp/amd64_rhel60/bin:/share/apps/idev/bin:/cm/shared/apps/slurm/14.03.0/sbin:/cm/shared/apps/slurm/14.03.0/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/ibutils/bin:/opt/dell/srvadmin/bin:/home/workshop/tuhpc002/bin [tuhpc002@cypress1 ~]$ }}} == bash_profile and bashrc == We can control the behavior of bash at login by editing the hidden configuration files in our home directory. {{{ [tuhpc002@cypress1 ~]$ ls -al .bash* -rw------- 1 tuhpc002 workshop 1259 Aug 18 22:34 .bash_history -rw-r--r-- 1 tuhpc002 workshop 18 Jul 18 2013 .bash_logout -rw-r--r-- 1 tuhpc002 workshop 176 Jul 18 2013 .bash_profile -rw-r--r-- 1 tuhpc002 workshop 124 Sep 30 2014 .bashrc [tuhpc002@cypress1 ~]$ }}} The files we wish to concentrate on are ".bash_profile" and ".bashrc". Let's examine their contents {{{ [tuhpc002@cypress1 ~]$ cat .bash_profile # .bash_profile # Get the aliases and functions if [ -f ~/.bashrc ]; then . ~/.bashrc fi # User specific environment and startup programs PATH=$PATH:$HOME/bin export PATH [tuhpc002@cypress1 ~]$ cat .bashrc # .bashrc # Source global definitions if [ -f /etc/bashrc ]; then . /etc/bashrc fi # User specific aliases and functions [tuhpc002@cypress1 ~]$ }}} You can see that the two are nearly identical. Aside from their reference to different directories (/etc vs ~), the only other difference is that our .bash_profile modifies and exports the PATH environment variable. There is much confusion over the role that each of these files play. When you login to a system, the .bash_profile file is executed UPON LOGIN. The .bashrc file modifies the default environment when connected without login, for example when executing commands remotely using SSH. Note that changes made in the .bashrc file will be reflected in the .bash_profile file but not vise versa. == SSH config (Linux and Mac)== It can be cumbersome to type your username and host address every time you login to a machine. The .ssh/config file offers the ability to create aliases to speed up this process. Add the following to your .ssh/config file to create a shortcut {{{ Host cypress1 Hostname cypress1.tulane.edu User }}} Now from your local machine you can simply type '''ssh cypress1''' and the username and hostname will be set automatically. === Next Section === [[cypress/ModuleCommand|Modules]]