A.2 Case Study: Semi-Partitioned Heterogeneous Cluster Dedicated to Parallel and Time-sharing Serial Jobs

Overview:

A site possessing a mixture of uniprocessor and dual processor nodes desires to dedicate a subset of nodes to time-sharing serial jobs, a subset to parallel batch jobs, and provide a set of nodes to be used as 'overflow'.

Resources:

    Compute Nodes:         Group A:    16 uniprocessor Linux based nodes, each with 128 MB of RAM and 1 GB local scratch space
                                           Group B:      8 2 way SMP Linux based nodes, each with 256 MB of RAM and 4 GB local scratch space
                                           Group C:      8 uniprocessor Linux based nodes, each with 192 MB of RAM and 2 GB local scratch space

Resource Manager: OpenPBS 2.3
Network: 100 MB switched ethernet

Workload:

Job Size: range in size from 1 to 32 processors with approximately the following quartile job frequency distribution
1 - 2, 3 - 8, 9 - 24, and 25 - 32 nodes.

Job Length: jobs range in length from 1 to 24 hours

Job Owners: job are submitted from 6 major groups consisting of a total of about 50 users

NOTES: During prime time hours, the majority of jobs submitted are smaller, short running development jobs where users are testing out new code and new data sets. The owners of these jobs are often unable to proceed with their work until a job they have submitted completes. Many of these jobs are interactive in nature. Throughout the day, large, longer running production workload is also submitted but these jobs do not have comparable turnaround time pressure.

Constraints: (Must do)

Nodes in Group A must run only parallel jobs. Nodes in Group B must only run serial jobs, with up to 4 serial jobs per node. Nodes in Group C must not be used unless a job cannot locate resources elsewhere.

Goals: (Should do)

The scheduler should attempt to intelligently load balance the timesharing nodes.

Analysis:

As in Case Study 1, The network topology is flat and and nodes are homogeneous within each group. The only tricky part of this configuration is the 'overflow' group. The easiest configuration is to create two PBS queues, serial and parallel, with appropriate min and max node counts as desired. By default, Maui interprets the PBS 'exclusive hostlist' queue attribute as constraining jobs in the queue to run only on the nodes contained in the hostlist. We can take advantage of this behavior to assign nodes in Group A and Group C to the queue 'parallel' while the nodes in Group B and Group C are assigned to the queue 'serial' (The same can be done with classes if using Loadleveler) Maui will incorporate this queue information when making scheduling decisions.

The next step is to make the scheduler use the 'overflow' nodes of group C only as a last resort. This can be accomplished using a negative affinity standing reservation. This configuration will tell the scheduler that these nodes can be used, but should only be used if it cannot find compute resources elsewhere.

The final step, load balancing, is accomplished in two parts. First, the nodes in group B must be configured to allow up to 4 serial jobs to run at a time. This is best accomplished using the PBS 'virtual nodes' feature. To load balance, simply select the CPULOAD allocation algorithm in Maui. This algorithm will instruct Maui to schedule the job based on which node has the most available, unused idle CPU time.

Configuration:

This site requires both resource manager and scheduler configuration.

The following Maui configuration would be needed.

maui.cfg
-----
# reserve 'overflow' processors

SRNAME[0] overflow
SRHOSTLIST[0] cn0[1-8] # hostname regular expression
SRCLASSLIST[0] parallel- batch- # use minus sign to indicate negative affinity

ALLOCATIONPOLICY CPULOAD

# allow SMP node sharing

NODEACCESSPOLICY SHARED
-----

PBS qmgr config
-----
set queue serial resources_max.nodeccount=1
set queue serial acl_hosts=an01+an02+...an16+cn01+cn02+...cn08
set queue serial acl_host_enable=true

set queue parallel resources_min.nodecount=2
set queue parallel acl_hosts=bn01+bn02+...bn08+cn01+cn02+...cn08
set queue parallel acl_host_enable=true
-----

PBS 'server_priv/nodes' file
-----
bn01 np=4
bn01 np=4
...
-----

Monitoring:

Conclusions:


	Learn More About Us Product Information Access Portal Demo Highlighted Customers Training Documentation News Download Moab Cluster Suite ·Customer Download ·Eval Download ·Access Portal Demo Moab Grid Suite ·Customer Download ·Eval Download Maui TORQUE Purchase Products Support Other Services Authorized Resellers Support Customer Support Purchase Support Evaluation Support Documentation Mailing List Contact Us Product Information Sales Technical Support Free Eval Support Partner Press
Home Products Services Resources Partners Press News Company	A.2 Case Study: Semi-Partitioned Heterogeneous Cluster Dedicated to Parallel and Time-sharing Serial Jobs Overview: A site possessing a mixture of uniprocessor and dual processor nodes desires to dedicate a subset of nodes to time-sharing serial jobs, a subset to parallel batch jobs, and provide a set of nodes to be used as 'overflow'. Resources: Compute Nodes: Group A: 16 uniprocessor Linux based nodes, each with 128 MB of RAM and 1 GB local scratch space Group B: 8 2 way SMP Linux based nodes, each with 256 MB of RAM and 4 GB local scratch space Group C: 8 uniprocessor Linux based nodes, each with 192 MB of RAM and 2 GB local scratch space Resource Manager: OpenPBS 2.3 Network: 100 MB switched ethernet Workload: Job Size: range in size from 1 to 32 processors with approximately the following quartile job frequency distribution 1 - 2, 3 - 8, 9 - 24, and 25 - 32 nodes. Job Length: jobs range in length from 1 to 24 hours Job Owners: job are submitted from 6 major groups consisting of a total of about 50 users NOTES: During prime time hours, the majority of jobs submitted are smaller, short running development jobs where users are testing out new code and new data sets. The owners of these jobs are often unable to proceed with their work until a job they have submitted completes. Many of these jobs are interactive in nature. Throughout the day, large, longer running production workload is also submitted but these jobs do not have comparable turnaround time pressure. Constraints: (Must do) Nodes in Group A must run only parallel jobs. Nodes in Group B must only run serial jobs, with up to 4 serial jobs per node. Nodes in Group C must not be used unless a job cannot locate resources elsewhere. Goals: (Should do) The scheduler should attempt to intelligently load balance the timesharing nodes. Analysis: As in Case Study 1, The network topology is flat and and nodes are homogeneous within each group. The only tricky part of this configuration is the 'overflow' group. The easiest configuration is to create two PBS queues, serial and parallel, with appropriate min and max node counts as desired. By default, Maui interprets the PBS 'exclusive hostlist' queue attribute as constraining jobs in the queue to run only on the nodes contained in the hostlist. We can take advantage of this behavior to assign nodes in Group A and Group C to the queue 'parallel' while the nodes in Group B and Group C are assigned to the queue 'serial' (The same can be done with classes if using Loadleveler) Maui will incorporate this queue information when making scheduling decisions. The next step is to make the scheduler use the 'overflow' nodes of group C only as a last resort. This can be accomplished using a negative affinity standing reservation. This configuration will tell the scheduler that these nodes can be used, but should only be used if it cannot find compute resources elsewhere. The final step, load balancing, is accomplished in two parts. First, the nodes in group B must be configured to allow up to 4 serial jobs to run at a time. This is best accomplished using the PBS 'virtual nodes' feature. To load balance, simply select the CPULOAD allocation algorithm in Maui. This algorithm will instruct Maui to schedule the job based on which node has the most available, unused idle CPU time. Configuration: This site requires both resource manager and scheduler configuration. The following Maui configuration would be needed. maui.cfg ----- `# reserve 'overflow' processors` `SRNAME[0] overflow` `SRHOSTLIST[0] cn0[1-8] # hostname regular expression` `SRCLASSLIST[0] parallel- batch- # use minus sign to indicate negative affinity` `ALLOCATIONPOLICY CPULOAD` `# allow SMP node sharing` `NODEACCESSPOLICY SHARED` ----- PBS qmgr config ----- `set queue serial resources_max.nodeccount=1` `set queue serial acl_hosts=an01+an02+...an16+cn01+cn02+...cn08` `set queue serial acl_host_enable=true` `set queue parallel resources_min.nodecount=2` `set queue parallel acl_hosts=bn01+bn02+...bn08+cn01+cn02+...cn08` `set queue parallel acl_host_enable=true` ----- PBS 'server_priv/nodes' file ----- `bn01 np=4` `bn01 np=4` `...` ----- Monitoring: Conclusions:
© 2001-2010 Adaptive Computing Enterprises, Inc.