A.15 Case Study: Cluster Consolidation

Overview

A large corporation has multiple clusters scattered within and across departments. This cluster fragmentation exists for three primary reasons:

Political - each department wants to manage and tune its own policies and priorities
Workload - some clusters are batch based, some are service or dedicated application based
Architectural - some technologies are not easily managed in a single integrated environment and thus clusters are divided along network or compute node boundaries

The corporation is finding significant costs associated with the duplication of office space, electrical, cooling, and IT staff resources and would like to reduce these costs.

Solution

Moab offers a solution which will allow resource consolidation while still maintaining department sovereignty and ease of administration.

Virtual Private Clusters

Many clusters are divided for political reasons allowing each sub-cluster to be independently controlled and configured according to local needs. A virtual private cluster (VPC) can be created for each department consisting of the following:

private compute nodes
private policies
private queue configuration
private set of users and accounts
private set of administrators

Each VPC can have complete autonomy and be independently managed and configured by an admin representing the particular needs of the associated department. Users within each VPC see only their own nodes, queues, jobs, and statistics, and are impacted only by their VPC's policies.

Virtual Jobs

Clusters are often created to address specific workload needs. Often, some clusters support mixed batch needs while others offer a persistent dedicated service such as a parallel database, a web farm, visualization services, or an accounting application. Using virtual jobs, Moab can encapsulate these dedicated services and allow them to peacefully co-exist with each other and even within shared mixed batch environment. The virtual job allows Moab to dynamically adjust resource allocation in a manner understood by the individual application so as to balance competing needs of the various workload types and better fulfill mission objectives. Using this same method, Moab is also able to provide monitoring, accounting, and capacity planning services and improve availability by re-allocating resources around failures. Further, as compute resources are no longer fragmented, organizations receive the benefits of load-balancing and consolidation.

Node Sets, Multi-RM and Peer to Peer

Some technologies are not compatible with others. This fact can be exhibited in 3 basic degrees:

nodes of differing configurations should not be allocated within the same job
differing technologies are best managed under different resource managers
differing technologies should be placed in independent clusters

Moab's nodeset feature enables an organization to allow efficient specification of desired heterogeneous resource usage to minimize co-allocation of diverse resource types. The specification can be applied on a per job, per queue, or per partition basis and tuned for maximum efficiency.

Moab's multi-resource manager feature allow resources from independent resource managers to be integrated into a single virtual cluster while allowing each resource type to be independently managed. This provides a single submission point for users and management point for administrators.

Moab's peer-to-peer facility allows even highly diverse resource types to be integrated into a unified compute resource by allowing single point submission for all users to the compute pool, allowing automatic job translation and job migration, and enabling credential and data mapping services.