Moab Workload Manager®

16.2 Testing New Middleware

Moab can be used to drive new middleware in essence stress testing resource management systems, information services, allocation services, security services, data staging services, and other aspects. Moab is unique when compared to other stress testing tools as it can perform the tests in response to actual or recorded workload traces, in essence performing a playback of events and driving the underlying system as if it were part of the production environment.

This feature can be used to identify scalability issues, pathological use cases, and accounting irregularities in anything from gridftp, to LDAP, to NIS and NFS.

Using Moab's time management facilities, Moab can drive the underlying systems in accordance with the real recorded distribution of time, at a multiplier of real time, or as fast as possible.

The following table describes some aspects of cluster analysis that can be driven by Moab.

System	Details
Allocation Manager	Use test or simulation mode to drive scheduling queries, allocation debits, and reservations to accounting packages. Verify synchronization of cluster statistics and stress test interfaces and underlying databases. (Set environment variable MOABAMTEST=yes to enable.)
Data Staging	Use test or simulation mode to drive real or virtual workload and manage associated data transfers to/from hierarchical storage systems, peer grid clusters, and other remote data repositories. Verify correct operation and stress test underlying network and storage hardware and data migration services. (Set environment variable MOABDSTEST=yes to enable.)
Identity Manager	Use test or simulation mode to query credential contact, QoS, fairshare, global usage, resource access, and relationship information and publish local usage information. Confirm interface protocol, business logic, and identity manager scalability. (Set environment variable MOABIMTEST=yes to enable.)
Information Services	Use test, simulation, or native resource manager mode to query information from and publish Moab event, state, and historical statistics to information services. Validate synchronization, scalability, and fault tolerance of information services. (Set environment variable MOABISTEST=yes to enable.)
Configuring a Grid with Globus	Use simulation or native resource manager mode to emulate a production cluster complete with workload queues. Drive job translation, job migration, and security layer facilities verifying end-to-end operation, reliability, responsiveness, and infrastructure scalability.
On-Demand/Provisioning Services	Use simulation or native resource manager mode to drive triggers and resource management interfaces to enable dynamic provisioning of hardware, operating systems, application software, and services. Test reliability and scalability of data servers, networks, and provisioning software as well as the interfaces and business logic coordinating these changes.
Resource Monitoring	Use test or native resource manager mode to actively load information from compute, network, storage, and software license managers confirming validity of data, availability during failures, and scalability.
Resource Management
Security Infrastructure
Arbitrary Middleware	Arbitrary middleware can be evaluated using Moab job profiles and object trigger facilities.

With each evaluation, the following tests can be enabled:

• functionality
• reliability
  • hard failure
    • hardware failure - compute, network, and data failures
    • software failure - loss of software services (NIS, LDAP, NFS, database)
    • soft failure
    • network delays, full file system, dropped network packets
  • corrupt data
• performance
• determine peak responsiveness in seconds/request
• determine peak throughput in requests/second
• determine responsiveness under heavy load conditions
• determine throughput under external load conditions
  • large user base (many users, groups, accounts)
  • large workload (many jobs)
  • large cluster (many nodes)
• manageability
  • full accounting for all actions/events
  • actions/failures can be easily and fully diagnosed

NOTE: If using a native resource manager and you do not want to actually submit real workload, you can set the environment variable MFORCESUBMIT to allow virtual workload to be managed without ever launching a real process.

General Analysis

For all middleware interfaces, Moab provides built-in performance analysis and failure reporting. Diagnostics for these interfaces are available via the mdiag command.

Simulation Based Analysis

Native Mode Analysis

Using native mode analysis, organizations can run Moab in normal mode with all facilities fully enabled, but with the resource manager fully emulated. With a native resource manager interface, any arbitrary cluster can be emulated with a simple script or flat text file. Artificial failures can be introduced, jobs can be virtually running, and artificial performance information generated and reported.

In the simplest case, emulation can be accomplished using the following configuration:

SCHEDCFG[natcluster] MODE=NORMAL SERVER=test1.bbli.com

ADMINCFG[1] USERS=dev

RMCFG[natcluster] TYPE=NATIVE CLUSTERQUERYURL=file://$HOME/cluster.dat

The preceding configuration will load cluster resource information from the file cluster.dat. An example resource information file follows:

node01 state=idle cproc=2
node02 state=idle cproc=2
node03 state=idle cproc=2
node04 state=idle cproc=2
node05 state=idle cproc=2
node06 state=idle cproc=2
node07 state=idle cproc=2
node08 state=idle cproc=2

In actual usage, any number of node attributes may be specified to customize these nodes, but in this example, only the node state and node configured processors attributes are specified.

The RMCFG flag NOSTART indicates that Moab should not actually issue a job start command to an external entity to start the job, but rather start the job logically internally only.

If it is desirable to take an arbitrary action at the start of a job, end of a job, or anywhere in between, the JOBCFG parameter can be used to create one or more arbitrary triggers to initiate internal or external events. The triggers can do anything from executing a script, to updating a database, to using a web service.

Using native resource manager mode, jobs may be introduced using the msub command according to any arbitrary schedule. Moab will load them, schedule them, and start them according to all site mission objectives and policies and drive all interfaced services as if running in a full production environment.