| Moab Workload Manager® |
13.5 Managing Resources Directly with the Native Interface
- 13.5.1 Native Interface Overview
- 13.5.2 Configuring the Native Interface
- 13.5.3 Generating Cluster Query Data
- 13.5.4 Configuring Node Specific Query URLs
- 13.5.5 Configuring Resource Types
- 13.5.6 Creating New Tools to Manage the Cluster
13.5.1 Native Interface Overview
The native interface allows a site to augment or even fully replace a resource manager for managing resources. In some situations, the full capabilities of the resource manager are not needed and a lower cost or lower overhead alternative is preferred. In other cases, the nature of the environment may make use of a resource manager impossible due to lack of support. Still, in other situations it is desirable to provide information about additional resource attributes, constraints, of state from alternate sources.
In any case, Moab provides the ability to directly query and manage resources along side of or without the use of a resource manager. This interface, called the NATIVE interface can also be used to launch, cancel, and otherwise manage jobs. This native interface offers several advantages including the following:
- No cost associated with purchasing a resource manager.
- No effort required to install or configure the resource manager.
- Ability to support abstract resources.
- Ability to support abstract jobs.
- Ability to integrate node availability information from multiple sources.
- Ability to augment node configuration and utilization information provided by a resource manager.
However, the NATIVE interface may also have some drawbacks:
- No support for standard job submission languages.
- Limited default configured and utilized resource tracking (additional resource tracking available with additional effort).
At a high level, the native interface works by launching threaded calls to perform standard resource manager activities such as managing resources and jobs. The desired calls are configured within Moab and used whenever an action or updated information is required.
13.5.2 Configuring the Native Interface
Using the native interface consists of defining the interface type and location. As mentioned earlier, a single object may be fully defined by multiple interfaces simultaneously with each interface updating a particular aspect of the object.
13.5.2.1 Configuring the Resource Manager
The native resource manager must be configured using the RMCFG parameter. To specify the native interface, the TYPE attribute must be set to NATIVE.
| SubType | Overview | Description |
| GANGLIA | ganglia | Use ganglia to monitor compute nodes. |
| SSH | secure shell extension | Use SSH facilities to monitor resources and manage jobs. |
| XCAT | Use the XCat toolset to monitor compute nodes, provision resources, and manage hardware. | |
| XT | Cray XT3 | Use low-level Cray system facilities to monitor and manage cluster resources. |
13.5.2.2 Reporting Resources
To indicate the source of the resource information, the CLUSTERQUERYURL attribute of the RMCFG parameter should be specified. This attribute is specified as a URL where the protocols FILE, EXEC, HTTP, GANGLIA, and SQL are allowed. If a protocol is not specified, the protocol EXEC is assumed.
| Format | Description |
| EXEC | Execute the script specified by the URL path. Use the script stdout as data. |
| FILE | Load the file specified by the URL path. Use the file contents as data. |
| GANGLIA | Query the Ganglia service located at the URL host and port. Directly process the query results using native Ganglia data formatting. |
| HTTP | Read the data specified by the URL. Use the raw data returned. |
| SQL | Load data directly from an SQL database using the FULL format described in what follows. |
This simple example queries a file on the server for information about every node in the cluster. This differs from Moab remotely querying the status of each node individually.
13.5.3 Generating Cluster Query Data
13.5.3.1 Flat Cluster Query Data
If the EXEC, FILE, or HTTP protocol is specified in the CLUSTERQUERYURL attribute, the data should provide flat text strings indicating the state and attributes of the node. The format follows the Wiki Interface Specification where attributes are delimited by white space rather than semicolons (;):
Describes any set of node attributes with format: <NAME> <ATTR>=<VAL> [<ATTR>=<VAL>]... | <NAME> - name of node <ATTR> - node attribute <VAL> - value of node attribute | n17 CPROC=4 AMEMORY=100980 STATE=idle |
13.5.3.2 Interfacing to MySQL
Moab can query a database to gather information about a cluster's compute node resources. An example configuration can be as follows:
Moab attempts to connect to the database MyNodes running on localhost as the user username. Once connected to MyNodes, Moab queries the table Nodes. This table must be set up to match the Full format just described. If the connection is successful, node information is populated based on the information found in the database.
13.5.3.3 Interfacing to Ganglia
Moab can use the information returned from Ganglia, a cluster or grid monitoring system. The information returned from Ganglia is combined with the information reported from other resource managers. An example configuration can be as follows:
<NodeName> is the name of a machine with Ganglia running on it. Also, <Port> is the XML port number to query Ganglia. If only ganglia:// is supplied as the CLUSTERQUERYURL, Moab queries the localhost on Ganglia's default port, 8649.
If Ganglia and Moab are running on different machines, the machine running Moab needs to be specified as a trusted host in Ganglia's configuration file.
Because Ganglia does not manage a job queue, Moab cannot control it or manage it; rather, Moab can only read in information. TORQUE reports nodes and can start jobs on those nodes. Moab and TORQUE can run concurrently without any issue because their functions do not overlap. However, it is mostly true that if Ganglia and TORQUE report conflicting data, you will want to trust TORQUE over Ganglia. For this reason you give the Ganglia resource manager the slavepeer flag. Also, Ganglia cannot report node state where state means availability to run jobs.
NOTE: To verify that Ganglia is correctly reporting information, issue the mdiag -R -v command or run telnet localhost 8649 and verify that appropriate XML is displayed.
13.5.3.4 Interfacing to FLEXlm
Moab can interface with FLEXlm to provide scheduling based on license availability. Informing Moab of license dependencies can reduce the number of costly licenses required by your cluster by allowing Moab to intelligently schedule around license limitations.
Provided with Moab in the tools directory is a Perl script, license.mon.FLEXlm.pl. This script queries a FLEXlm license server and gathers data about available licenses. This script then formats this data for Moab to read through a native interface. This script can easily be used by any site to help facilitate FLEXlm integration; the only modification necessary to the script is setting the $FLEXlmCmd to specify the local command to query FLEXlm. To make this change, edit license.mon.FLEXlm.pl and, near the top of the file, look for the following line:
Set the $FLEXlmCmd to the appropriate value for your system to query a license server and license file (if applicable). If lmutil is not in the PATH variable, specify its full path. Using lmutil's -a argument causes it to report all licenses. The -c option can be used to specify an optional license file.
To test this script, run it manually. If working correctly, it produces output similar to the following:
If the output looks incorrect, set the $LOGLEVEL variable inside of license.mon.FLEXlm.pl, run it again, and address the reported failure.
Once the license interface script is properly configured, the next step is to add a license native resource manager to Moab via the moab.cfg file:
Once this change is made, restart Moab. The command mdiag -R can be used to verify that the resource manager is properly configured and is in the Active state. Detailed information regarding configured and utilized licenses can be viewed by issuing the mdiag -n. Floating licenses (non-node-locked) are reported as belonging to the GLOBAL node.
NOTE: Due to the inherent conflict with the plus sign (+), the provided license manager script replaces occurrences of the plus sign in license names with the underscore symbol (_). This replacement requires that licenses with a plus sign in their names be requested with an underscore in place of any plus signs.
Interfacing to Multiple License Managers Simultaneously
If multiple license managers are used within a cluster, Moab can interface to each of them to obtain the needed license information. In the case of FLEXlm, this can be done by making one copy of the license.mon.FLEXlm.pl script for each license manager and configuring each copy to point to a different license manager. Then, within Moab, create one native resource manager interface for each license manager and point it to the corresponding script as in the following example:
NOTE: For an overview of license management, including job submission syntax, see Section 13.7, License Management.
NOTE: It may be necessary to increase the default limit, MMAX_GRES. (See Appendix D for more implementation details.)
13.5.4 Configuring Node Specific Query URLs
It is possible to have a separate CLUSTERQUERYURL for each node. This is possible using the NODECFG parameter for each node or for the DEFAULT node. Moab looks first on the specific node for CLUSTERQUERYURL information. If no information is found on the specific node it looks for CLUSTERQUERYURL information on the resource manager. If the resource manager has no query information specified, then it uses the CLUSTERQUERYURL command configured for the DEFAULT node.
The example configuration that follows demonstrates a possible setup:
In the preceding example, a four-node cluster and a license manager are controlled via the native interface. The state of the four compute nodes is determined by running the /usr/local/bin/query.pl query command (remotely on the node) and specified within the DEFAULT NODECFG parameter while querying the license manager is accomplished using the /usr/local/bin/flquery.cgi script. For local executable scripts, the launched script is either locally generated or taken from the library of contributed native scripts included with the distribution file.
An optional parameter, RESOURCELIST, may be specified to constrain which resources obtained by the native interface should be processed. By default, all resources described by the interface data are loaded for *FULL interfaces, as are all existing resources described by *EXT interfaces. The RESOURCELIST parameter, if specified, acts as a filter eliminating either full or extension resource information from being incorporated. If an environment exists where data is not aggregated, and the native interface provides primary node information, the RESOURCELIST parameter is required to indicate to Moab which resources should be in the cluster.
13.5.5 Configuring Resource Types
Native resource managers can also perform special tasks when they are given a specific resource type. These types are specified using the RESOURCETYPE attribute of the RMCFG parameter.
| TYPE | EXPLANATION |
| COMPUTE | Normal compute resources. (No special handling.) |
| FS | File system resource manager. (See Multiple Resource Managers for an example.) |
| LICENSE | Software license manager. (See Interfacing with FLEXlm and License Management.) |
| NETWORK | Network resource manager. |
13.5.6 Creating New Tools to Manage the Cluster
Using the scripts found in the $TOOLSDIR ($INSTDIR/tools) directory as a template, new tools can be quickly created to monitor or manage most any resource. Each tool should be associated with a particular resource manager service and specified using one of the following resource manager URL attributes:
|
SERVICE | CLUSTERQUERY |
| ATTRIBUTE | CLUSTERQUERYURL |
| DETAILS | Queries resource state, configuration, and utilization information for compute nodes, networks, storage systems, software licenses, and other resources. |
| INPUT | --- |
| OUTPUT | Node status and configuration for one or more nodes. (See Resource Data Format.) |
| EXAMPLE |
Moab executes the storquery.pl script to obtain information about 'v-stor' resources. |
|
SERVICE | JOBMODIFY |
| ATTRIBUTE | JOBMODIFYURL |
| DETAILS | Modified a job or application. |
| INPUT | [-j <JOBEXPR>] [--s[et]|--c[lear]|--i[ncrement]|--d[ecrement]] <ATTR>[=<VALUE>] [<ATTR>[=<VALUE>]]... |
| OUTPUT | --- |
| EXAMPLE |
Moab executes the jobmodify.pl script to modify the specified job. |
|
SERVICE | JOBRESUME |
| ATTRIBUTE | JOBRESUMEURL |
| DETAILS | Resumes a suspended job or application. |
| INPUT | <JOBID> |
| OUTPUT | --- |
| EXAMPLE |
Moab executes the jobresume.pl script to resume suspended jobs. |
|
SERVICE | JOBSTART |
| ATTRIBUTE | JOBSTARTURL |
| DETAILS | Launches a job or application on a specified set of resources. |
| INPUT | <JOBID> <TASKLIST> <USERNAME> [ARCH=<ARCH>] [OS=<OPSYS>] [IDATA=<STAGEINFILEPATH>[,<STAGEINFILEPATH>]...] [EXEC=<EXECUTABLEPATH>] |
| OUTPUT | --- |
| EXAMPLE |
Moab executes the jobstart.pl script to execute jobs. |
|
SERVICE | JOBSUSPEND |
| ATTRIBUTE | JOBSUSPENDURL |
| DETAILS | Suspends in memory an active job or application. |
| INPUT | <JOBID> |
| OUTPUT | --- |
| EXAMPLE |
Moab executes the jobsuspend.pl script to suspend active jobs. |
|
SERVICE | JOBVALIDATE |
| ATTRIBUTE | JOBVALIDATEDURL |
| DETAILS | Validates acceptability of newly submitted job. If the validate URL reports a failure status code, then the associated job will be rejected. The specific rejection action can be controlled by the job validate tool if it reports the attribute, ACTION={cancel|requeue|suspend}. If success is reported, the validate tool can modify the job attributes listed in the OUTPUT section that follows. This is done by having the tool report a series of attribute-value pairs with one pair per line using the WIKI job description language: WCLIMIT=600\nQOS=low\n. |
| INPUT | [UNAME=<USERNAME>] [EUSER=<USERNAME>] [GNAME=<GROUPNAME>] [WCLIMIT=<WCLIMIT>] [RCLASS=<CLASSNAME>] [QOS=<QOSNAME>] [EXEC=<COMMAND>] [RFEATURES=<FEATURES>] [VARIABLE=<VARIABLES>] [GRES=<GRES>] |
| OUTPUT | [ACCOUNT=<ACCOUNTNAME>\n][RCLASS=<CLASSNAME>\n][QOS=<QOSNAME>\n][RFEATURES=<FEATURES>\n][WCLIMIT=<WCLIMIT>\n][VARIABLE=<VARIABLES>\n] |
| EXAMPLE |
Moab executes the jobvalidate.pl script to validate each newly submitted job. |
|
SERVICE | NODEMODIFY |
| ATTRIBUTE | NODEMODIFYURL |
| DETAILS | Provide method to dynamically modify/provision compute resources including operating system, applications, queues, node features, power states, and so forth. |
| INPUT | <NODEID>[,<NODEID>] [--force] {--set <ATTR>=<VAL>|--clear <ATTR>} ATTR is one of the node attributes listed in Resource Data Format |
| OUTPUT | -- |
| EXAMPLE |
Moab reprovisions compute nodes using the provision.pl script. |
|
SERVICE | RMINITIALIZE |
| ATTRIBUTE | RMINITIALIZEURL |
| DETAILS | Provide method to initialize new interface with native RM. NOTE: This interface is called only once at scheduler start up before any other RM interface is activated. |
| INPUT | --type=<TYPE> --replace=<VARIABLES> [<OPTIONAL_ARGS>]... |
| OUTPUT | N/A |
| EXAMPLE | Moab will initialize the ccs interface by executing the win.init.pl script with the appropriate command-line arguments. |
|
SERVICE | SYSTEMMODIFY |
| ATTRIBUTE | SYSTEMMODIFYURL |
| DETAILS | Provide method to dynamically modify aspects of the compute environment that are directly associated with cluster resources. |
| INPUT | ??? |
| OUTPUT | <STRING> |
| EXAMPLE | Moab modifies high-level aspects of the cluster environment using the specified script. |
|
SERVICE | SYSTEMQUERY |
| ATTRIBUTE | SYSTEMQUERYURL |
| DETAILS | Provide method to dynamically query aspects of the compute environment that are directly associated with cluster resources. |
| INPUT | default <ATTR> ATTR is one of images |
| OUTPUT | <STRING> |
| EXAMPLE | Moab loads the list of images available from Warewulf using the checkimage.pl script. |
|
SERVICE | WORKLOADQUERY |
| ATTRIBUTE | WORKLOADQUERYURL |
| DETAILS | Provide method to dynamically query the system workload (such as jobs and services) of the compute environment that are associated with managed resources. NOTE: Job/workload information should be reported back from the URL (script, file, webservice, and so forth) using the WIKI job description language. |
| INPUT | --- |
| OUTPUT | <STRING> |
| EXAMPLE | Moab loads job/workload information by executing the job.query.xt3.pl script. |
|
|
See Also
- mdiag -R command (evaluate resource managers)
- License Management
- Peer to Peer Management