Moab Workload Manager Administrator's Guide

Overview    The Moab Workload Manager is a policy engine which allows sites control over when, where, and how resources such as processors, memory, and disk space are allocated to jobs.  In addition to this control, it also provides mechanisms which intelligently optimize the use of these resources, monitor system performance, help diagnose problems, and generally manage the system.

• 1.0  Philosophy and Goals of the Moab Workload Manager
  • 1.1  Value of a Batch System
  • 1.2  Philosophy and Goals
• 2.0  Installation and Initial Configuration
  • 2.1  Building and Installing Moab
  • 2.2  Initial Configuration
  • 2.3  Initial Testing (Monitor, Interactive, Simulation and Normal Modes)
• 3.0  Scheduler Basics
  • 3.1  Layout of Scheduler Components
  • 3.2  Scheduling Environment and Objects
  • 3.3  Scheduling Iterations and Job Flow
  • 3.4  Configuring the Scheduler
  • 3.5  Credential Overview
• 4.0  Scheduler Commands
  • 4.1  Client Overview
  • 4.2  Monitoring System Status
  • 4.3  Managing Jobs
  • 4.4  Managing Reservations
  • 4.5  Configuring Policies
  • 4.6  End User Commands
  • 4.7  Miscellaneous Commands
• 5.0  Prioritizing Jobs and Allocating Resources
  • 5.1  Job Priority
  • 5.2  Node Allocation
  • 5.3  Node Access
  • 5.4  Node Availability
  • 5.5  Task Distribution
  • 5.6  Virtualization and Resource Provisioning
• 6.0  Managing Fairness - Throttling Policies, Fairshare, and Allocation Management
  • 6.1  Fairness Overview
  • 6.2  Usage Limits/Throttling Policies
  • 6.3  Fairshare
  • 6.4  Charging and Allocation Management
  • 6.5  Internal Charging Facilities
• 7.0  Controlling Resource Access - Reservations, Partitions, and QoS Facilities
  • 7.1  Advance Reservations
  • 7.2  Partitions
  • 7.3  QoS Facilities
• 8.0  Optimizing Scheduling Behavior - Backfill, Node Sets, and Preemption
  • 8.1  Optimization Overview
  • 8.2  Backfill
  • 8.3  Node Sets
  • 8.4  Preemption
• 9.0  Statistics, Accounting, and Profiling
  • 9.1  Scheduler Performance Evaluation Overview
  • 9.2  Accounting - Job and System Statistics
  • 9.3  Testing New Versions and Configurations
  • 9.4  Answering 'What If?' Questions with the Simulator
• 10.0  Managing Shared Resources - Usage Based Limits, Policies, and SMP Issues
  • 10.1  Consumable Resource Handling
  • 10.2  Load Balancing Features
  • 10.3  Resource Usage Tracking
  • 10.4  Resource Usage Limits
  • 10.5  General SMP Issues
• 11.0  General Job Administration
  • 11.1  Deferred Jobs and Job Holds
  • 11.2  Job Priority Management
  • 11.3  Suspend/Resume Handling
  • 11.4  Checkpoint/Restart
  • 11.5  Job Dependencies
  • 11.6  Setting Job Defaults and Per Job Limits
  • 11.7  General Job Policies
  • 11.8  Using a Local Queue
  • 11.9  Job Deadline Support
  • 11.10  Job Templates
• 12.0  General Node Administration
  • 12.1  Node Location (Partitions, Racks, Queues, etc.)
  • 12.2  Node Attributes (Node Features, Speed, etc.)
  • 12.3  Node Specific Policies (MaxJobPerNode, etc.)
  • 12.4  Configuring Node-Locked Consumable Generic Resources (tape drives, node-locked licenses, etc.)
  • 12.5  Managing Shared Cluster Resources (shared filesystems, licenses, etc.)
  • 12.6  Node State Management
• 13.0  Resource Managers and Interfaces
  • 13.1  Resource Manager Overview
  • 13.2  Resource Manager Configuration
  • 13.3  Resource Manager Extensions
  • 13.4  Adding Resource Manager Interfaces
  • 13.5  Managing Resources Directly with the Native Interface
  • 13.6  Utilizing Multiple Resource Managers
  • 13.7  License Management
  • 13.8  Provisioning Managers
  • 13.9  Network Management
  • 13.10  Intelligent Platform Management Interface (IPMI)
• 14.0  Troubleshooting and System Maintenance
  • 14.1  Internal Diagnostics
  • 14.2  Logging Facilities
  • 14.3  Object Messages
  • 14.4  Notifying Administrators of Failures and Critical Events
  • 14.5  Issues with Client Commands
  • 14.6  Tracking System Failures
  • 14.7  Problems with Individual Jobs
  • 14.8  Diagnostic Scripts
• 15.0  Improving User Effectiveness
  • 15.1  User Feedback Loops
  • 15.2  User Level Statistics
  • 15.3  Enhancing Wallclock Limit Estimates
  • 15.4  Providing Resource Availability Information
  • 15.5  Job Start Time Estimates
  • 15.6  Collecting Performance Information on Individual Jobs
• 16.0  Cluster Analysis, Testing, and Simulation
  • 16.1  Evaluating New Releases and Policies
  • 16.2  Testing New Middleware
  • 16.3  Evaluating New Workload and Resources
  • 16.4  Simulation Overview
  • 16.5  Analyzing Long Term Impacts of Potential Changes
• 17.0  Peer To Peer
  • 17.1  Peer To Peer (P2P) Basics
  • 17.2  Initial Peer Configuration
  • 17.3  P2P Resource Information and Access Control
  • 17.4  P2P Data Management
  • 17.5  P2P Security and Credential Management
  • 17.6  P2P Usage Policies
  • 17.7  P2P Resource Affinity
  • 17.8  P2P General Management and Diagnostics
  • 17.9  P2P Tuning and Optimization
  • 17.10  P2P Hierarchies
  • 17.11  Coordinating Shared Resource Access w/P2P
  • 17.12  Grid Peer To Peer
  • 17.13  P2P Grids and Globus
  • 17.14  P2P Grid Rollout, Testing, and Troubleshooting
  • 17.15  P2P Accounting and Allocation Management
• 18.0  Data Staging
  • 18.1  Data Staging Overview
• 19.0  Utility Computing with Moab (Hosting on Demand)
  • 19.1  Utility Computing Overview
  • 19.2  Resource Monitoring in a Utility Computing Hosting Center
  • 19.3  Resource Allocation and Environment Management
  • 19.4  Provisioning Resources and Managing Dynamic Security
  • 19.5  Service Level Agreements and QoS Guarantees
  • 19.6  Tight Integration with Customer Resources
  • 19.7  Enabling Manual Resource Access Requests
  • 19.8  Automating Resource Access Requests
  • 19.9  Standby Resources
  • 19.10  Accounting, Costing, and Automated Billing
  • 19.11  Utility Computing Example: Setting up a Test Center in 1 Hour or Less
• 20.0  Object Triggers
  • 20.1  Object Triggers Overview
• 21.0  Virtual Private Clusters
  • 21.1  Virtual Private Clusters Overview
• 22.0  Miscellaneous
  • 22.1  User Feedback
  • 22.2  Enabling High Availability Features
  • 22.3  Using the Application Scheduling Library
  • 22.4  Dynamic and Malleable Jobs
  • 22.5  Identity Managers
  • 22.6  Information Services for the Enterprise and Grid
• Appendices
  • Appendix A:  Case Studies
  • Appendix B:  Extension Interface
  • Appendix C:  Adding New Algorithms
  • Appendix D:  Adjusting Default Limits
  • Appendix E:  Security Configuration
  • Appendix F:  Parameters Overview
  • Appendix G:  Commands Overview
  • Appendix H:  Interfacing with Moab
  • Appendix I:  Considerations for Large Clusters
  • Appendix J:  Adding Moab as a Service
  • Appendix K:  Migrating from Maui 3.2
  • Appendix L:  Moab Java API
  • Appendix M:  Moab Differences Guide
  • Appendix N:  Moab-Maui Comparison Guide
  • Appendix O:  Resource Manager Integration Overview