13.5 Managing Resources Directly with the Native Interface
In any case, Moab provides the ability to directly query and manage resources along side of or without the use of a resource manager. This interface, called the NATIVE interface can also be used to launch, cancel, and otherwise manage jobs. This NATIVE interface offers several advantages including the following:
However, the NATIVE interface may also have some drawbacks.
At a high level, the native interface works by launching threaded calls to perform standard resource manager activities such as managing resources and jobs. The desired calls are configured within Moab and used whenever an action or updated information is required.RMCFG parameter. To specify the native interface, the TYPE attribute must be set to NATIVE.
RMCFG[local] TYPE=NATIVE RMCFG[local] CLUSTERQUERYURL=exec:///tmp/query.sh
RMCFG parameter should be specified. This attribute is specified as a URL where the protocols FILE, EXEC, HTTP, GANGLIA, and SQL are allowed. If a protocol is not specified, the protocol EXEC is assumed.
Moab considers a NativeRM script to have failed if it returns with a non-zero exit code, or if the CHILDSTDERRCHECK parameter is set and its appropriate conditions are met. In addition, the NativeRM script associated with a job submit URL will be considered as having failed if its standard output stream contains the text, "ERROR".
This simple example queries a file on the server for information about every node in the cluster. This differs from Moab remotely querying the status of each node individually.
RMCFG[local] TYPE=NATIVE RMCFG[local] CLUSTERQUERYURL=file:///tmp/query.txt
RMCFG[TORQUE] TYPE=pbs RMCFG[ganglia] TYPE=NATIVE CLUSTERQUERYURL=ganglia://<NodeName>:<Port> RMCFG[ganglia] FLAGS=SLAVEPEER NODESTATEPOLICY=OPTIMISTIC
<NodeName> is the name of a machine with Ganglia running on it. Also, <Port> is the xml port number to query Ganglia. If only ganglia:// is supplied as the CLUSTERQUERYURL, Moab will query the localhost on Ganglia's default port, 8649.
If Ganglia and Moab are running on different machines, the machine running Moab needs to be specified as a trusted host in Ganglia's configuration file.
Because Ganglia is not a real resource manager, in that it does not manage a job queue, Moab cannot control it or manage it, it can only read in information. TORQUE is a real resource manager in that it reports nodes and can start jobs on those nodes. The two can run concurrently without any issue, because their "responsiblities" do not overlap. However, it is mostly true that if Ganglia and TORQUE report conflicting data, you will want to trust TORQUE over Ganglia. For this reason you give the Ganglia RM the "slave" flag. Also, Ganglia cannot report node "state" where state means "availability to run jobs."
The following list of Ganglia metrics are supported up to Ganglia version 3.1.1:
Moab can interface with FLEXlm to provide scheduling based on license availability. Informing Moab of license dependencies can reduce the number of costly licenses required by your cluster by allowing Moab to intelligently schedule around license limitations.
Provided with Moab in the tools directory is a Perl script, license.mon.flexLM.pl. This script queries a FLEXlm license server and gathers data about available licenses. This script then formats this data for Moab to read through a native interface. This script can easily be used by any site to help facilitate FLEXlm integration--the only modification necessary to the script is setting the $FLEXlmCmd to specify the local command to query FLEXlm. To make this change, edit license.mon.flexLM.pl and, near the top of the file, look for the line:
my $FLEXlmCmd = "SETME";
Set the '$FLEXlmCmd' to the appropriate value for your system to query a license server and license file (if applicable). If lmutil is not in the PATH variable, specify its full path. Using lmutil's -a argument will cause it to report all licenses. The -c option can be used to specify an optional license file.
To test this script, run it manually. If working correctly, it will produce output similar to the following:
> ./license.mon.flexLM.pl GLOBAL UPDATETIME=1104688300 STATE=idle ARES=autoCAD:130,idl_mpeg:160 CRES=autoCAD:200,idl_mpeg:330
If the output looks incorrect, set the $LOGLEVEL variable inside of license.mon.flexLM.pl, run it again, and address the reported failure.
Once the license interface script is properly configured, the next step is to add a license native resource manager to Moab via the moab.cfg file:
RMCFG[FLEXlm] TYPE=NATIVE RESOURCETYPE=LICENSE RMCFG[FLEXlm] CLUSTERQUERYURL=exec://$TOOLSDIR/license.mon.flexLM.pl ...
Once this change is made, restart Moab. The command mdiag -R can be used to verify that the resource manager is properly configured and is in the state Active. Detailed information regarding configured and utilized licenses can be viewed by issuing the mdiag -n. Floating licenses (non-node-locked) will be reported as belonging to the GLOBAL node.
Interfacing to Multiple License Managers Simultaneously
If multiple license managers are used within a cluster, Moab can interface to each of them to obtain the needed license information. In the case of FLEXlm, this can be done by making one copy of the license.mon.flexLM.pl script for each license manager and configuring each copy to point to a different license manager. Then, within Moab, create one native resource manager interface for each license manager and point it to the corresponding script as in the following example:
RMCFG[FLEXlm1] TYPE=NATIVE RESOURCETYPE=LICENSE RMCFG[FLEXlm1] CLUSTERQUERYURL=exec://$TOOLSDIR/license.mon.flexLM1.pl RMCFG[FLEXlm2] TYPE=NATIVE RESOURCETYPE=LICENSE RMCFG[FLEXlm2] CLUSTERQUERYURL=exec://$TOOLSDIR/license.mon.flexLM2.pl RMCFG[FLEXlm3] TYPE=NATIVE RESOURCETYPE=LICENSE RMCFG[FLEXlm3] CLUSTERQUERYURL=exec://$TOOLSDIR/license.mon.flexLM3.pl ...
Nagios installation and configuration documentation can be found at Nagios.org.
Provided with Moab in the tools directory is a Perl script, node.query.nagios.pl. This script reads the Nagios status.dat file and gathers data about network hosts and services. This script then formats data for Moab to read through a native interface. This script can be used by any site to help facilitate Nagios integration. To customize the data that will be formatted for Moab, make the changes in this script.
You may need to customize the associated configuration file in the etc directory, config.nagios.pl. The statusFile line in this script tells Moab where the Nagios status.dat file is located. Make sure that the path name specified is correct for your site. Note that the interval which Nagios updates the Nagios status.dat file is specified in the Nagios nagios.cfg file. Refer to Nagios documentation for further details.
To make these changes, familiarize yourself with the format of the Nagios status.dat file and make the appropriate additions to the script to include the desired Wiki Interface attributes in the Moab output.
To test this script, run it manually. If working correctly, it will produce output similar to the following:
> ./node.query.nagios.pl gateway STATE=Running localhost STATE=Running CPULOAD=1.22 ADISK=75332
Once the Nagios interface script is properly configured, the next step is to add a Nagios native resource manager to Moab via the moab.cfg file:
RMCFG[nagios] TYPE=NATIVE RMCFG[nagios] CLUSTERQUERYURL=exec://$TOOLSDIR/node.query.nagios.pl ...
Once this change is made, restart Moab. The command mdiag -R can be used to verify that the resource manager is properly configured and is in the state Active. Detailed information regarding configured Nagios node information can be viewed by issuing the mdiag -n -v.
> mdiag -n -v compute node summary Name State Procs Memory Disk Swap Speed Opsys Arch Par Load Rsv Classes Network Features gateway Running 0:0 0:0 0:0 0:0 1.00 - - dav 0.00 0 - - - WARNING: node 'gateway' is busy/running but not assigned to an active job WARNING: node 'gateway' has no configured processors localhost Running 0:0 0:0 75343:75347 0:0 1.00 - - dav 0.48 0 - - - WARNING: node 'localhost' is busy/running but not assigned to an active job WARNING: node 'localhost' has no configured processors ----- --- 3:8 1956:1956 75345:75349 5309:6273 Total Nodes: 2 (Active: 2 Idle: 0 Down: 0)
RMCFG[TORQUE] TYPE=pbs RMCFG[supermon] TYPE=NATIVE CLUSTERQUERYURL=exec://$HOME/tools/node.query.supermon.pl
To confirm that Supermon is properly connected to Moab, issue "mdiag -R -v." The output should be similar to the following example, specifically there are no errors about the CLUSTERQURYURL.
diagnosing resource managers RM[TORQUE] State: Active Type: PBS ResourceType: COMPUTE Server: keche Version: '2.2.0-snap.200707181818' Job Submit URL: exec:///usr/local/bin/qsub Objects Reported: Nodes=3 (6 procs) Jobs=0 Flags: executionServer Partition: TORQUE Event Management: EPORT=15004 (no events received) Note: SSS protocol enabled Submit Command: /usr/local/bin/qsub DefaultClass: batch RM Performance: AvgTime=0.26s MaxTime=1.04s (4 samples) RM Languages: PBS RM Sub-Languages: - RM[supermon] State: Active Type: NATIVE:AGFULL ResourceType: COMPUTE Cluster Query URL: exec://$HOME/node.query.supermon.pl Objects Reported: Nodes=3 (0 procs) Jobs=0 Partition: supermon Event Management: (event interface disabled) RM Performance: AvgTime=0.03s MaxTime=0.11s (4 samples) RM Languages: NATIVE RM Sub-Languages: - Note: use 'mrmctl -f messages ' to clear stats/failures
Run the Perl script by itself. The script's results should look similar to this:
vm01 GMETRIC[CPULOAD]=0.571428571428571 GMETRIC[NETIN]=133 GMETRIC[NETOUT]=702 GMETRIC[NETUSAGE]=835 vm02 GMETRIC[CPULOAD]=0.428571428571429 GMETRIC[NETIN]=133 GMETRIC[NETOUT]=687 GMETRIC[NETUSAGE]=820 keche GMETRIC[CPULOAD]=31 GMETRIC[NETIN]=5353 GMETRIC[NETOUT]=4937 GMETRIC[NETUSAGE]=10290
If the preceeding functioned properly, issue a checknode command on one of the nodes that Supermon is gathering statistics for. The output should look similiar to below.
node keche State: Idle (in current state for 00:32:43) Configured Resources: PROCS: 2 MEM: 1003M SWAP: 3353M DISK: 1M Utilized Resources: --- Dedicated Resources: --- Generic Metrics: CPULOAD=33.38,NETIN=11749.00,NETOUT=9507.00,NETUSAGE=21256.00 MTBF(longterm): INFINITY MTBF(24h): INFINITY Opsys: linux Arch: --- Speed: 1.00 CPULoad: 0.500 Network Load: 0.87 kB/s Flags: rmdetected Network: DEFAULT Classes: [batch 2:2][interactive 2:2] RM[TORQUE]: TYPE=PBS EffNodeAccessPolicy: SHARED Total Time: 2:03:27 Up: 2:03:27 (100.00%) Active: 00:00:00 (0.00%) Reservations: ---
Native resource managers using HTTP URLs send and receive information using the standard HTTP 1.0 protocol. Information is sent using the HTTP GET method, while results are to be returned in the HTTP body using the format described in the Flat Cluster Query Data section. Not all available native resource manager query URLs are currently supported. Following is a chart showing the supported query URLs and the parameters that will be provided by MOAB in the GET request.
CGI scripts pointed to by the query URLs should always return at least one line of output on success to insure that Moab does not consider empty result sets to be a failure. In the case of empty result sets, this can be accomplished by returning an empty comment line (i.e., the '#' character followed by a newline).
It is possible to have a separate CLUSTERQUERYURL for each node. This is possible using the NODECFG parameter for each node or for the DEFAULT node. Moab will look first on the specific node for CLUSTERQUERYURL information. If no information is found on the specific node it will look for CLUSTERQUERYURL information on the Resource Manager. If the Resource Manager has no query information specified then it will use the CLUSTERQUERYURL command configured for the DEFAULT node.
The example configuration below demonstrates a possible setup.
RMCFG[local] TYPE=NATIVE RESOURCELIST node1,node2,node3,node4,flexlm1 NODECFG[DEFAULT] CLUSTERQUERYURL=exec:///usr/local/bin/query.pl NODECFG[flexlm1] CLUSTERQUERYURL=http://supercluster.org/usr/local/flquery.cgi
In the example above, a four node cluster and a license manager are controlled via the native interface. The state of the four compute nodes will be determined by running the /usr/local/bin/query.pl query command (remotely on the node) specified within the DEFAULT NODECFG parameter while querying the license manager will be accomplished using the /usr/local/bin/flquery.cgi script. For local executable scripts, the launched script is either locally generated or taken from the library of contributed native scripts included with the distribution file.
As above, an optional parameter, RESOURCELIST, may be specified to constrain which resources obtained by the native interface should be processed. By default, all resources described by the interface data are loaded. The RESOURCELIST parameter, if specified, acts as a filter eliminating either full or extension resource information from being incorporated. If an environment exists where data is not aggregated, and the native interface provides primary node information, the RESOURCELIST parameter is required to indicate to Moab which resources should be in the cluster.
Using the scripts found in the $TOOLSDIR ($INSTDIR/tools) directory as a template, new tools can be quickly created to monitor or manage most any resource. Each tool should be associated with a particular resource manager service and specified using one of the following resource manager URL attributes.
Searches Moab documentation only
|© 2001-2010 Adaptive Computing Enterprises, Inc.|