[an error occurred while processing this directive] [an error occurred while processing this directive] [an error occurred while processing this directive]

A.2  Case Study:   Semi-Partitioned Heterogeneous Cluster Dedicated to Parallel and Time-sharing Serial Jobs


    A site possessing a mixture of uniprocessor and dual processor nodes desires to dedicate a subset of nodes to time-sharing serial jobs, a subset to parallel batch jobs, and provide a set of nodes to be used as 'overflow'.


    Compute Nodes:         Group A:    16  uniprocessor Linux based nodes, each with 128 MB of RAM and 1 GB local scratch space
                                           Group B:      8  2 way SMP Linux based nodes, each with 256 MB of RAM and 4 GB local scratch space
                                           Group C:      8  uniprocessor Linux based nodes, each with 192 MB of RAM and 2 GB local scratch space

    Resource Manager:    OpenPBS 2.3
    Network:                      100 MB switched ethernet


    Job Size:                range in size from 1 to 32 processors with approximately the following quartile job frequency distribution
                                         1 - 2, 3 - 8, 9 - 24, and 25 - 32 nodes.

    Job Length:           jobs range in length from 1 to 24 hours

    Job Owners:         job are submitted from 6 major groups consisting of a total of about 50 users

    NOTES:              During prime time hours, the majority of jobs submitted are smaller, short running development jobs where users are testing out new code and new data sets.  The owners of these jobs are often unable to proceed with their work until a job they have submitted completes.  Many of these jobs are interactive in nature.  Throughout the day, large, longer running production workload is also submitted but these jobs do not have comparable turnaround time pressure.

Constraints: (Must do)

    Nodes in Group A must run only parallel jobs.  Nodes in Group B must only run serial jobs, with up to 4 serial jobs per node.  Nodes in Group C must not be used unless a job cannot locate resources elsewhere.

Goals: (Should do)

    The scheduler should attempt to intelligently load balance the timesharing nodes.


    As in Case Study 1, The network topology is flat and and nodes are homogeneous within each group.   The only tricky part of this configuration is the 'overflow' group.  The easiest configuration is to create two PBS queues, serial and parallel, with appropriate min and max node counts as desired.  By default, Maui interprets the PBS 'exclusive hostlist' queue attribute as constraining jobs in the queue to run only on the nodes contained in the hostlist.  We can take advantage of this behavior to assign nodes in Group A and Group C to the queue 'parallel'  while the nodes in Group B and Group C are assigned to the queue 'serial' (The same can be done with classes if using Loadleveler)  Maui will incorporate this queue information when making scheduling decisions.

    The next step is to make the scheduler use the 'overflow' nodes of group C only as a last resort.  This can be accomplished using a negative affinity standing reservation.  This configuration will tell the scheduler that these nodes can be used, but should only be used if it cannot find compute resources elsewhere.

    The final step, load balancing, is accomplished in two parts.  First, the nodes in group B must be configured to allow up to 4 serial jobs to run at a time.  This is best accomplished using the PBS 'virtual nodes' feature.   To load balance, simply select the CPULOAD allocation algorithm in Maui.  This algorithm will instruct Maui to schedule the job based on which node has the most available, unused idle CPU time.


    This site requires both resource manager and scheduler configuration.

    The following Maui configuration would be needed.

# reserve 'overflow' processors

SRNAME[0]       overflow
SRHOSTLIST[0]   cn0[1-8]           # hostname regular expression
SRCLASSLIST[0]  parallel-  batch-  # use minus sign to indicate negative affinity


# allow SMP node sharing


PBS qmgr config
set queue serial resources_max.nodeccount=1
set queue serial acl_hosts=an01+an02+...an16+cn01+cn02+...cn08
set queue serial acl_host_enable=true

set queue parallel resources_min.nodecount=2
set queue parallel acl_hosts=bn01+bn02+...bn08+cn01+cn02+...cn08
set queue parallel acl_host_enable=true

PBS 'server_priv/nodes' file
bn01  np=4
bn01  np=4



[an error occurred while processing this directive] [an error occurred while processing this directive]