= Linux Commands =

If you have not done yet, '''download Sample files''' by:

{{{git clone https://hidekiCCS:@bitbucket.org/hidekiCCS/hpc-workshop.git}}}

== Basic Linux Commands == 
Here is a curated list of the most common Linux commands you will use. This list is by no means extensive, but these commands will allow you to navigate the clusters as well as create, destroy, and manipulate files and directories.

=== man ===
Display online manual pages. Most Linux commands have a manual page with detailed instructions on use. Replace '''<command name>''' below with command you would like information on.

{{{
[tulaneID@cypress1 ~]$ man <command name>
}}}

=== pwd ===
On a *nix system, directories are containers for files and objects. The '''pwd''' command lists the present working directory. This is the "where am I?" command.
{{{
[tulaneID@cypress1 ~]$ pwd
/home/tulaneID
[tulaneID@cypress1 ~]$
}}}

=== cd ===

The '''cd''' command changes directory. (Observe the change of directory name in the resulting system prompt as well!)

{{{
[tulaneID@cypress1 ~]$ cd hpc-workshop
[tulaneID@cypress1 hpc-workshop]$ pwd
/home/tulaneID/hpc-workshop
[tulaneID@cypress1 hpc-workshop]$
}}}

As you can see, workshop is a ''subdirectory'' of tulaneID. And tulaneID is itself a subdirectory of home, which is a subdirectory of the root directory (/). Thus, the directories form a downward facing tree with all directories stemming from the root directory. You can move one level up in the tree by typing '''cd ..''

{{{
[tulaneID@cypress1 hpc-workshop]$ pwd
/home/tulaneID/workshop
[tulaneID@cypress1 hpc-workshop]$ cd ..
[tulaneID@cypress1 ~]$ pwd
/home/tulaneID
[tulaneID@cypress1 ~]$
}}}

And you can navigate using either relative or absolute paths. That is, you can enter directory paths relative to your current location, or you can entire the entire path starting with the root directory.

{{{
[tulaneID@cypress1 ~]$ pwd
/home
[tulaneID@cypress1 ~]$ cd tulaneID/hpc-workshop
[tulaneID@cypress1 hpc-workshop]$ pwd
/home/tulaneID/hpc-workshop
[tulaneID@cypress1 ~]$ cd /home
[tulaneID@cypress1 ~]$ pwd
/home
[tulaneID@cypress1 ~]$
}}}

Lastly, if you ever get lost you can use the tilde (~ - or simply **cd** with no argument!) to return to your HOME directory.
{{{
[tulaneID@cypress1 ~]$ cd ~
[tulaneID@cypress1 ~]$ pwd
/home/tulaneID
[tulaneID@cypress1 ~]$
}}}


=== ls ===
The '''ls''' command will list files in the current directory.
{{{
[tulaneID@cypress1 ~]$ ls
hpc-workshop
[tulaneID@cypress1 ~]$
}}}

If you would like to see if a directory contains a specific file, you can pass the directory path and file name to '''ls''' as an argument
{{{
[tulaneID@cypress1 ~]$ ls hpc-workshop
BlasLapack  Eigen3        HeatMass    JobArray1  JobDependencies  MPI     PETSc  precision  Python  ScaLapack  SimpleExample  TestCodes  VTK
CUDA        FlowInCavity  hybridTest  JobArray2  Matlab           OpenMP  PI     PSE        R       SerialJob  SLU40          uBLAS
[tulaneID@cypress1 ~]$
}}}

Setting the {{{-l}}} flag will display files along with their permissions and ownership.
{{{
[tulaneID@cypress1 ~]$ ls -l hpc-workshop
total 100
drwxr-xr-x 3 fuji hpcstaff 4096 Aug 21 13:06 BlasLapack
drwxr-xr-x 3 fuji hpcstaff 4096 Aug 21 13:06 CUDA
drwxr-xr-x 3 fuji hpcstaff 4096 Aug 21 13:06 Eigen3
drwxr-xr-x 3 fuji hpcstaff 4096 Aug 21 13:06 FlowInCavity
drwxr-xr-x 3 fuji hpcstaff 4096 Aug 21 13:06 HeatMass
drwxr-xr-x 3 fuji hpcstaff 4096 Aug 21 13:06 hybridTest
drwxr-xr-x 3 fuji hpcstaff 4096 Aug 21 14:23 JobArray1
drwxr-xr-x 3 fuji hpcstaff 4096 Aug 21 14:42 JobArray2
drwxr-xr-x 3 fuji hpcstaff 4096 Aug 21 15:20 JobDependencies
drwxr-xr-x 3 fuji hpcstaff 4096 Aug 21 13:06 Matlab
drwxr-xr-x 3 fuji hpcstaff 4096 Aug 21 13:06 MPI
drwxr-xr-x 3 fuji hpcstaff 4096 Aug 21 13:06 OpenMP
drwxr-xr-x 3 fuji hpcstaff 4096 Aug 21 13:06 PETSc
drwxr-xr-x 3 fuji hpcstaff 4096 Aug 21 13:06 PI
drwxr-xr-x 3 fuji hpcstaff 4096 Aug 21 13:06 precision
drwxr-xr-x 3 fuji hpcstaff 4096 Aug 21 13:06 PSE
drwxr-xr-x 3 fuji hpcstaff 4096 Aug 21 13:15 Python
drwxr-xr-x 3 fuji hpcstaff 4096 Aug 21 13:06 R
drwxr-xr-x 3 fuji hpcstaff 4096 Aug 21 13:06 ScaLapack
drwxr-xr-x 3 fuji hpcstaff 4096 Aug 21 13:45 SerialJob
drwxr-xr-x 9 fuji hpcstaff 4096 Aug 21 13:06 SimpleExample
drwxr-xr-x 5 fuji hpcstaff 4096 Aug 21 13:06 SLU40
drwxr-xr-x 3 fuji hpcstaff 4096 Aug 21 13:06 TestCodes
drwxr-xr-x 3 fuji hpcstaff 4096 Aug 21 13:06 uBLAS
drwxr-xr-x 3 fuji hpcstaff 4096 Aug 21 13:06 VTK
[tulaneID@cypress1 ~]$
}}}

=== mkdir ===
Create new directory.
{{{
[tulaneID@cypress1 ~]$ cd testdir
-bash: cd: testdir: No such file or directory
[tulaneID@cypress1 ~]$ mkdir testdir
[tulaneID@cypress1 ~]$ cd testdir
[tulaneID@cypress1 testdir]$ pwd
/home/tulaneID/testdir
[tulaneID@cypress1 testdir]$
}}}

=== rm ===
Remove files and directories. CAUTION: THERE IS NO UNDOING THIS COMMAND.
{{{
[tulaneID@cypress1 ~]$ rm hpc-workshop
rm: cannot remove `hpc-workshop': Is a directory
[tulaneID@cypress1 ~]$
}}}
If you would like to remove a directory and all of its contents - with prompts for each directory and subdirectory - use the following command:
{{{
[tulaneID@cypress1 ~]$ rm -ri hpc-workshop
rm: descend into directory `hpc-workshop'? y
rm: descend into directory `hpc-workshop/PI'? n
rm: descend into directory `hpc-workshop/uBLAS'? ^C
[tulaneID@cypress1 ~]$ 
}}}
Use Ctrl+c to exit the prompts.

The {{{-r}}} flag specifies to remove files/directories recursively. {{{-i}}} specifies to prompt before deleting each file.

If you are confident with the command line and your understanding of file locations you can use the {{{-f}}} flag instead of {{{-i}}} to force deletion of all files without prompting:
{{{
[tulaneID@cypress1 ~]$ rm -rf hpc-workshop
[tulaneID@cypress1 ~]$ ls hpc-workshop
ls: cannot access workshop: No such file or directory
[tulaneID@cypress1 ~]$
}}}

Do this again to download sample files:

{{{git clone https://hidekiCCS:@bitbucket.org/hidekiCCS/hpc-workshop.git}}}

== rmdir ==
Removes directories. CAUTION: THERE IS NO UNDOING THIS COMMAND AS WELL.

The '''rmdir''' is a safer approach than using '''rm -r''' as it will only act on empty directories.

{{{
[tulaneID@cypress1 ~]$ ls NextDirectoryDown/
file.txt
[tulaneID@cypress1 ~]$ rmdir NextDirectoryDown/
rmdir: failed to remove `NextDirectoryDown/': Directory not empty
[tulaneID@cypress1 ~]$ rm NextDirectoryDown/file.txt 
[tulaneID@cypress1 ~]$ rmdir NextDirectoryDown/
[tulaneID@cypress1 ~]$ ls
[tulaneID@cypress1 ~]$ 

}}}

=== cp ===
Copy file to new location.  CAUTION: THERE IS NO UNDOING THIS COMMAND AS WELL - SEE ALSO OPTIONS {{{-i}}} and {{{-n}}}.

In the example below '''sourcefile''' already exists, and '''destinationfile''' may or may not already exist and will become a copy of '''sourcefile'''.
{{{
[tulaneID@cypress1 ~]$ cp sourcefile destinationfile
}}}
To copy a directory and all of its contents we use the recursive flag {{{-r}}}:
{{{
[tulaneID@cypress1 ~]$ cp -r sourcedir destinationdir
}}}

Take note of the order here: source first, destination second. This is the standard order across most *nix commands.
=== mv ===
Move a file to a new location. CAUTION: THERE IS NO UNDOING THIS COMMAND AS WELL - SEE ALSO OPTIONS {{{-i}}} and {{{-n}}}.
{{{
[tulaneID@cypress1 ~]$ mv file1 file2
[tulaneID@cypress1 ~]$ ls file1
ls: cannot access file1: No such file or directory
[tulaneID@cypress1 ~]$ ls file2
file2
[tulaneID@cypress1 ~]$
}}}
Unlike '''cp''', the '''mv''' command does not create a second instance of the file or directory.

=== cat ===
Print the entire contents of a file (short for conCATenate).
{{{
[tulaneID@cypress1 ~]$ cat daysofweek.txt
monday
tuesday
wednesday
thursday
friday
saturday
sunday
[tulaneID@cypress1 ~]$
}}}

=== tail ===
Displays lines from the end of a file. Useful for viewing recent results in an output file.
{{{
[tulaneID@cypress1 ~]$ cat animals
dog
Dog
cat
Racoon
DOG
bullfrog
Little Doggie

[tulaneID@cypress1 ~]$ tail -3 animals
DOG
bullfrog
Little Doggie
[tulaneID@cypress1 ~]$
}}}
The {{{-f}}} flag can be used to tail a file interactively. New additions to the end of the file will be printed to your screen.

=== head ===
'''head''' is much like '''tail''', except it prints from the top of a file.
{{{
[tulaneID@cypress1 ~]$ head -2 animals
dog
Dog
[tulaneID@cypress1 ~]$
}}}

=== more ===
'''more''' is like '''cat''', except it prints the file one page at a time. The spacebar is used to continue on to the next page.

=== less ===
'''less''' is like '''more''', except it handles large files more efficiently in memory and supports a bit more flexibility in file navigation.

=== zmore, zless ===
'''zmore''' and '''zless''' are the counterparts of '''more''' and '''less''', respectively, for viewing files (with extension '''.gz''') compressed via '''gzip'''.

(See also '''man gzip'''.)

=== which ===
'''which <command name>''' tells the location of the named command.
{{{
[tulaneID@cypress1 ~]$ which zless
/usr/bin/zless
[tulaneID@cypress1 ~]$
}}}

== Basic Linux Commands Exercise ==

Lets take a break from lecture to practice some of the commands we've just learned

[[cypress/BasicLinuxComands/Exercise|Basic Commands Exercise]]

== Not So Basic Linux Commands ==

=== Star Wildcard ===
Shell commands can often make use of the wildcard characters. The most universal of these is the asterisk or star wildcard, *. This acts as a stand-in for any and all strings. 
 
=== ls again ===
Suppose we have a directory containing many files, but we only want to list files ending with the .txt extension. We can use the * wildcard to request a list of any file ending in .txt

{{{
[tulaneID@cypress1 ~]$ ls
a.out  NextDirectoryDown helloworld.c  textfile01.txt  textfile02.txt
[tulaneID@cypress1 ~]$ ls *.txt
textfile01.txt  textfile02.txt
[tulaneID@cypress1 ~]$ 
}}}

Additionally, the behavior of the '''ls''' command can be modified by the addition of option flags. By default '''ls''' does not list any "hidden" files and subdirectories. To display all files and subdirectories one must add the {{{-a}}} flag.

{{{
[tulaneID@cypress1 ~]$ ls -l
total 4
-rw-r--r-- 1 tulaneID workshop    0 Aug 18 21:50 a.out
drwxr-xr-x 2 tulaneID workshop 4096 Aug 18 21:37 NextDirectoryDown
-rw-r--r-- 1 tulaneID workshop    0 Aug 18 21:50 helloworld.c
-rw-r--r-- 1 tulaneID workshop    0 Aug 18 21:50 textfile01.txt
-rw-r--r-- 1 tulaneID workshop    0 Aug 18 21:50 textfile02.txt
[tulaneID@cypress1 ~]$ ls -al
total 56
drwx------ 6 tulaneID workshop 4096 Aug 18 20:03 .
drwxr-xr-x 5 root     root     4096 Aug 12 13:00 ..
-rw------- 1 tulaneID workshop  107 Aug 18 16:59 .bash_history
-rw-r--r-- 1 tulaneID workshop   18 Jul 18  2013 .bash_logout
-rw-r--r-- 1 tulaneID workshop  176 Jul 18  2013 .bash_profile
-rw-r--r-- 1 tulaneID workshop  124 Sep 30  2014 .bashrc
-rw-r--r-- 1 tulaneID workshop  500 May  7  2013 .emacs
drwxr-xr-x 2 tulaneID workshop 4096 Nov 11  2010 .gnome2
-rw-r--r-- 1 tulaneID workshop  171 Aug  6  2014 .kshrc
drwxr-xr-x 4 tulaneID workshop 4096 Aug  6  2014 .mozilla
drwxr-xr-x 2 tulaneID workshop 4096 Aug 18 16:17 NextDirectoryDown
drwx------ 2 tulaneID workshop 4096 Aug 13 16:18 .ssh
-rw------- 1 tulaneID workshop 1786 Aug 18 20:03 .viminfo
-rw------- 1 tulaneID workshop   54 Aug 13 16:18 .Xauthority
[tulaneID@cypress1 ~]$ 
}}}

=== quota ===
Display disk quotas.

The {{{-s}}} flag translates the output into a readable format. If your blocks column is equal to or greater than the quota column, you have exceeded your available disk space.
{{{
[tulaneID@cypress1 ~]$quota -s
Disk quotas for user tulaneID (uid 12345): 
     Filesystem  blocks   quota   limit   grace   files   quota   limit   grace
master.10ge.cluster:/home
                  6459M  10000M  10000M           23260    250k    250k        
master.10ge.cluster:/share/apps
                  6459M  10000M  10000M           23260    250k    250k      
[tulaneID@cypress1 ~]$ 
}}}

There are two storage spaces for users on cypress, [[cypress/about#Storage:homedirectory|home directory]] and [[cypress/about#Storage:Lustregroupprojectdirectory|Lustre group project directory]].

=== grep ===
Find text in a file. '''grep''' will return every line in the file that matches your search term.
{{{
[tulaneID@cypress1 ~]$ cat animals
dog
Dog
cat
Racoon
DOG
bullfrog
Little Doggie
[tulaneID@cypress1 ~]$ grep dog animals
dog
[tulaneID@cypress1 ~]$ 
}}}
Use the {{{-v}}} flag to print out every line excluding those containing the search term. Notice that "dog" (lowercase) is missing from the results.
{{{
[tulaneID@cypress1 ~]$ grep -v dog animals
Dog
cat
Racoon
DOG
bullfrog
Little Doggie
[tulaneID@cypress1 ~]$ 
}}}
The {{{-i}}} flag will search without case-sensitivity.
{{{
[tulaneID@cypress1 ~]$ grep -i dog animals
dog
Dog
DOG
Little Doggie
[tulaneID@cypress1 ~]$ 
}}}
The {{{-i}}} and {{{-v}}} flags can be combined to exclude all lines containing the search term regardless of capitalization.
{{{
[tulaneID@cypress1 ~]$ grep -iv dog animals
cat
Racoon
bullfrog
[tulaneID@cypress1 ~]$ 
}}}

=== find ===
Searches for files in a directory hierarchy. The '''find''' takes a directory as an argument and will search in that directory and all of it's subdirectories for files that match the search criteria. If no directory is specified, the current directory is used.

You can search for a file by name using the {{{-name}}} flag as your search criteria 
{{{
[tulaneID@cypress1 ~]$ find /home/tulaneID/ -name file1.txt
/home/tulaneID/dir1/file1.txt
[tulaneID@cypress1 ~]$ 
}}}

One can also search for all files that have a string of text in any portion of their name by using the star wildcard, e.g. we can find all the .txt files in our HOME directory and it's subdirectories. Note that quotes are required around our criteria as we are now using a wildcard.

{{{
[tulaneID@cypress1 ~]$ find /home/tulaneID/ -name "*.txt"
/home/tulaneID/textfile02.txt
/home/tulaneID/dir1/file1.txt
/home/tulaneID/examples/alphabet.txt
/home/tulaneID/examples/numbers.txt
/home/tulaneID/textfile01.txt
[tulaneID@cypress1 ~]$ 
}}}

These are just a couple examples of the many ways '''find''' can be used. A nice short tutorial on '''find''' with a lot of examples can be found at http://alvinalexander.com/unix/edu/examples/find.shtml


=== Pipe ( | ) ===

The pipe command, '''|''', takes the output of one command and passes it to another. For example we could "pipe" the output of '''cat''' to '''grep''' in order to search a file for a particular string

{{{
[tulaneID@cypress1 examples]$ head -5 numbers.txt | grep 1
01
[tulaneID@cypress1 examples]$ 
}}} 

=== I/O Redirection ( >, >> and < ) ===

Redirect input or output. CAUTION: THERE IS NO UNDOING THE REDIRECT OVERWRITE `>` COMMAND.

The redirect output operator, `>`, takes the output (see '''man stdout''') of a command and redirects it to a file. For example we could "redirect" the output of '''grep''' above to  a file in order to capture the results of the above search

{{{
[tulaneID@cypress1 examples]$ head -5 numbers.txt | grep 1 > grep.out
[tulaneID@cypress1 examples]$ cat grep.out
01
}}}

The redirect append output operator `>>` does the same as `>` except the former APPENDS whereas the latter OVERWRITES the output to the specified file.

The redirect input operator `<` is used for redirecting input

{{{
[tulaneID@cypress1 examples]$ cat < grep.out
01
}}}

=== history ===
Displays the command history list with line numbers. You can add an optional integer, n, argument to display only the last n commands

{{{
[tulaneID@cypress1 examples]$ history 2
  299  cat numbers.txt | grep 1
  300  history 2
[tulaneID@cypress1 examples]$ 
}}}

Note: '''history''', '''|''', and '''grep''' when used together are a powerful combination as they allow you to search for old commands that you may not remember the exact syntax for:

{{{
[tulaneID@cypress1 examples]$ history | grep chmod
  153  chmod u+x myscript.sh 
  204  chmod +x myexecutable.sh 
  271  chmod +x myscript.sh 
  273  chmod -x myscript.sh 
  301  history | grep chmod
[tulaneID@cypress1 examples]$ 
}}}

=== chmod  ===
The '''chmod''' commands allow a user to modify the permissions of files and directories that they own. To see the permissions of a file/directory we can use the {{{ls -l}}} command

{{{
[tulaneID@cypress1 ~]$ ls -l
total 4
-rw-r--r-- 1 tulaneID workshop    0 Aug 18 21:50 a.out
-rw-r--r-- 1 tulaneID workshop    0 Aug 18 21:50 helloworld.c
-rw-r--r-- 1 tulaneID workshop    0 Aug 19 07:15 myscript.sh
drwxr-xr-x 2 tulaneID workshop 4096 Aug 18 21:37 NextDirectoryDown
-rw-r--r-- 1 tulaneID workshop    0 Aug 18 21:50 textfile01.txt
-rw-r--r-- 1 tulaneID workshop    0 Aug 18 21:50 textfile02.txt
[tulaneID@cypress1 ~]$
}}}

The ten left most characters describe your permissions. The first character describes the file type (- for a file, d for a directory). The next three describe the permissions for the user who owns the file, the middle three describe the permissions for the group that owns the file, and the last three describe the permissions for "others". The "r" denotes permission to read, the "w" means permission to write, and the "x" indicates permission to execute the file/directory.

The change mode or '''chmod''' command allows us to alter those permissions. In the past one had to remember (or google) numeric codes that corresponded to permission states, but modern syntax allows for the addition and subtraction of permissions with greater ease. Now one needs only indicate whose permissions to change and what changes to make. For example, let's add executable permissions for the user to the file myscript.sh

{{{
[tulaneID@cypress1 ~]$ chmod u+x myscript.sh 
[tulaneID@cypress1 ~]$ ls -l
total 4
-rw-r--r-- 1 tulaneID workshop    0 Aug 18 21:50 a.out
-rw-r--r-- 1 tulaneID workshop    0 Aug 18 21:50 helloworld.c
-rwxr--r-- 1 tulaneID workshop    0 Aug 19 07:15 myscript.sh
drwxr-xr-x 2 tulaneID workshop 4096 Aug 18 21:37 NextDirectoryDown
-rw-r--r-- 1 tulaneID workshop    0 Aug 18 21:50 textfile01.txt
-rw-r--r-- 1 tulaneID workshop    0 Aug 18 21:50 textfile02.txt
[tulaneID@cypress1 ~]$
}}}

The user (and only the user) can now run the script myscript.sh as an executable. For a more complete introduction see http://alvinalexander.com/linux-unix/linux-chmod-command-permissions-file-directories


= Next Section =
[[cypress/FileEditingSoftware|File editing software]]