| Moab Workload Manager® |
A.27 Case Study: Node Board Scheduling with PBSPro 9.x and SGI Altix
Overview
Sites that have PBSPro 9.x and SGI Altix systems may find Moab node board scheduling capabilities useful. Moab can take a request of CPUs and memory and determine how many node boards are required for the job. For example, a user may request one processor and 14 GB of memory (qsub -l ncpus=1,mem=14gb script) on a node that has four processors and 8 GB of memory per node board. Moab would determine that the job would need two node boards to fulfill the request.
Configuration
RMCFG[pbs] PBSALTIX=TRUE RMCFG[pbs] USEVNODES=TRUE RMCFG[pbs] IGNHNODES=TRUE NODESETPOLICY ONEOF NODESETATTRIBUTE FEATURE NODESETISOPTIONAL FALSE NODESETPRIORITYTYPE MINLOSS NODESETLIST cnode1,cnode2,cnode3,cnode4 NODESETDELAY 1:00 NODESETMAXUSAGE .5 NODEALLOCATIONPOLICY PRIORITY NODEACCESSPOLICY SINGLEJOB NODECFG[cnode1[1]] FEATURES=cnode1 NODECFG[cnode1[2]] FEATURES=cnode1 NODECFG[cnode1[3]] FEATURES=cnode1 NODECFG[cnode1[4]] FEATURES=cnode1 NODECFG[cnode2[1]] FEATURES=cnode2 NODECFG[cnode2[2]] FEATURES=cnode2 NODECFG[cnode2[3]] FEATURES=cnode2 NODECFG[cnode2[4]] FEATURES=cnode2 NODECFG[cnode2[1]] FEATURES=cnode3 NODECFG[cnode2[2]] FEATURES=cnode3 NODECFG[cnode2[3]] FEATURES=cnode3 NODECFG[cnode2[4]] FEATURES=cnode3 NODECFG[cnode2[1]] FEATURES=cnode4 NODECFG[cnode2[2]] FEATURES=cnode4 NODECFG[cnode2[3]] FEATURES=cnode4 NODECFG[cnode2[4]] FEATURES=cnode4 SRCFG[rsv1] PERIOD=INFINITY SRCFG[rsv1] HOSTLIST=l:cnode1[1],cnode1[2],cnode1[3],cnode1[4] SRCFG[rsv1] CLASSLIST=admin SRCFG[rsv1] MAXJOB=1
An smp node is represented in Moab as a NODESET. Each node board in the node needs to be associated with a NODESET, which is done by setting the FEATURES on the node to the NODESET's name. NODESETPOLICY, NODESETATTRIBUTE, NODESETPRIORITYTYPE, and NODESETISOPTIONAL must all be set to the corresponding values indicated in the previous configuration for node board scheduling to work. NODESETDELAY is available for node board scheduling.
NODEALLOCATIONPOLICY must be set to PRIORITY.
Standing reservations can be used to group the virtual nodes of a node so that policies can set on the smp node. For example, MAXJOB=1 can be used to limit only one running job on the smp node at a time. When listing virtual nodes in the the standing reservation's HOSTLIST l: must be used.
Note: mdiag -n -v can be used to view the total statistics of each host node gathered by Moab.
compute node summary Name State Procs Memory Disk Swap Speed Opsys Arch Par Load Rsv smp1[1] Idle 2:2 11264:11264 0:0 0:0 1.00 - - loc 0.00 0 smp1[2] Idle 2:2 11264:11264 0:0 0:0 1.00 - - loc 0.00 0 smp1[3] Idle 2:2 11264:11264 0:0 0:0 1.00 - - loc 0.00 0 smp2[1] Running 1:2 0:4096 0:0 0:0 1.00 - - loc 1.00 1 smp2[2] Idle 2:2 4096:4096 0:0 0:0 1.00 - - loc 0.00 0 smp2[3] Idle 2:2 4096:4096 0:0 0:0 1.00 - - loc 0.00 0 smp3[1] Running 1:4 0:8129 0:0 0:0 1.00 - - loc 3.00 1 smp3[2] Idle 4:4 8129:8129 0:0 0:0 1.00 - - loc 0.00 0 smp3[3] Idle 4:4 8129:8129 0:0 0:0 1.00 - - loc 0.00 0 smp summary smp1 6:6 33792:33792 smp2 4:6 8192:12288 smp3 8:12 16258:24387 ----- --- 20:24 58242:70467 0:0 0:0 Total Nodes: 9 (Active: 2 Idle: 7 Down: 0)