| 56 | | 2. Login to Cypress in a separate terminal session, and use the SLURM '''squeue''' command to determine the job's node list - in this case an idle interactive session. |
| 57 | | |
| 58 | | {{{ |
| 59 | | [tulaneID@cypress1 ~]$squeue -u $USER |
| 60 | | JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) |
| 61 | | 3272175 centos7 idv38428 tulaneID R 3:54 1 cypress01-059 |
| 62 | | }}} |
| 63 | | 3. For each job node, use the '''ssh''' and '''top''' commands in combination to determine the job's core usage on the given node such as the following. (See '''man top'''.) |
| 64 | | |
| 65 | | Here are the relevant output columns for the '''top''' command. |
| 66 | | |
| 67 | | ||='''top''' command output column=||='''Description'''=||='''Notes'''=|| |
| 68 | | ||%CPU||percentage of cores used per job process (100% per full core used)||sum(%CPU)/100=fractional # of cores in use on the node|| |
| 69 | | ||%MEM||percentage of RAM used per job process||sum(%MEM)=percentage of node's total RAM in use|| |
| 70 | | |
| 71 | | Here's the combined command and result. |
| | 56 | 2. '''Login to Cypress''' in a separate terminal session, and use the locally provided command, '''seff <job-ID>''', to determine the the usage and efficiency of the job's CPU and memory. |
| 74 | | [tulaneID@cypress1 ~]$ssh cypress01-059 top -b -n 1 -u $USER |
| 75 | | top - 00:34:01 up 75 days, 9:23, 1 user, load average: 0.06, 0.05, 0.01 |
| 76 | | Tasks: 730 total, 1 running, 729 sleeping, 0 stopped, 0 zombie |
| 77 | | Cpu(s): 52.7%us, 0.7%sy, 0.0%ni, 46.6%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st |
| 78 | | Mem: 66013252k total, 3374764k used, 62638488k free, 137060k buffers |
| 79 | | Swap: 12582904k total, 0k used, 12582904k free, 1280248k cached |
| 80 | | |
| 81 | | PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND |
| 82 | | 31502 tulaneID 20 0 27880 1772 956 R 3.8 0.0 0:00.09 top |
| 83 | | 30927 tulaneID 20 0 9200 1244 1044 S 0.0 0.0 0:00.00 slurm_script |
| 84 | | 30953 tulaneID 20 0 4072 544 464 S 0.0 0.0 0:00.00 sleep |
| 85 | | 30970 tulaneID 20 0 144m 2368 1164 S 0.0 0.0 0:00.00 sshd |
| 86 | | 30971 tulaneID 20 0 25092 3100 1516 S 0.0 0.0 0:00.06 bash |
| 87 | | 31501 tulaneID 20 0 144m 2316 1132 S 0.0 0.0 0:00.00 sshd |
| 88 | | |
| | 59 | [tulaneID@cypress1 ~]$seff 3336903 |
| | 60 | Job ID: 3336903 |
| | 61 | Cluster: cypress |
| | 62 | User/Group: cbaribault/hpcstaff |
| | 63 | State: RUNNING (exit code 0:0) |
| | 64 | Cores: 20 |
| | 65 | CPU Utilized: 0:00:00 |
| | 66 | CPU Efficiency: 0.0% of 00:02:26 core-walltime |
| | 67 | Memory Utilized: 0.00 GB |
| | 68 | Memory Efficiency: 0.0% of requested per-CPU memory x 20 |
| | 69 | ------------------------------------------------------------ |
| 90 | | |
| 91 | | 4. Next we'll re-run the same combined '''ssh...top...''' command and pipe the input to '''awk''' in order to sum the values in the columns %CPU, %MEM. |
| 92 | | |
| 93 | | {{{ |
| 94 | | [tulaneID@cypress1 ~]$ssh cypress01-065 'top -b -n 1 -u $USER' | \ |
| 95 | | awk 'NR > 7 { sum_cpu += $9; sum_mem += $10 } \ |
| 96 | | END { print "Total %CPU:", sum_cpu; print "Total %MEM:", sum_mem }' |
| 97 | | Total %CPU: 3.8 |
| 98 | | Total %MEM: 0 |
| 99 | | |
| 100 | | }}} |
| 101 | | |
| 102 | | 5. If the idev session requested 20 cores (default=20), then the core efficiency of the idle idev session is |
| 103 | | |
| 104 | | {{{ |
| 105 | | [tulaneID@cypress1 ~]$bc <<< "scale=4;(3.8 / 100) / 20" |
| 106 | | .0019 |
| 107 | | }}} |
| 108 | | |
| 109 | | This is quite far from the ideal value, 1 - not very good usage of the node's 20 requested cores. |
