Appendix H: Interfacing with Moab
Moab Workload Manager®

Appendix H: Interfacing with Moab

Moab provides numerous interfaces allowing it to monitor and manage most services and resources. It also possesses flexible interfaces to allow it to interact with peer services and applications as both a broker and an information service. This appendix is designed to provide a general overview and links to more detailed interface documentation.

Moab interfaces to systems providing various services and using various protocols. This appendix is designed to assist users who want to enable Moab in new environments using one of the existing interfaces. It does not cover the steps required to create a new interface.

H.1 Query and Control APIs

The Moab Cluster and Grid Suites provide a (Moab) workload manager server that supports a broad array of client services. These services can either be directly accessed via Moab client commands or by way of a number of APIs. Which approach is best will depend on the particular use case.

H.1.1 C API

The Moab C API provides access to all Moab information and services. This API consists of both easy to use high-level calls and powerful and flexible low-level calls.

High-Level C API Calls

free initialize jobcancel jobcheckpoint jobgetres jobgettime
joblistres jobmigrate jobmodify jobquery jobrequeue jobresume
jobsignal jobstart jobsubmit jobsuspend nodemodify

Function Args Return Code Description
MCCInitialize(Host,Port,&C) Host: (char *) optional server hostname or NULL
Port: (int) optional server port or 0
C: (void **) opaque interface handle 		Returns 0 on success or -1 on failure This call initializes the interface between the client and the server. It must be called once before attempting to issue any Moab client calls.
MCCJobCancel(C,JID,EMsg) C: (void **) address of interface handle
JID: (char *) job id to cancel (optional)
EMsg: (char *) optional error message 		Returns 0 on success or -1 on failure This call will cancel the specified job. If failures occur and EMsg is provided, this buffer will be populated with a human-readable error message.
MCCJobCheckpoint(C,JID,EMsg) C: (void **) address of interface handle
JID: (char *) job id to cancel (optional)
EMsg: (char *) optional error message 		Returns 0 on success or -1 on failure This call will checkpoint the specified job. If failures occur and EMsg is provided, this buffer will be populated with a human-readable error message.
MCCJobGetNumAllocRes(C,JID,Count,EMsg) C: (void **) address of interface handle
JID: (char *) job id to query (optional)
Count: (int *) output value (number of allocated hosts)
EMsg: (char *) optional error message 		Returns 0 on success or -1 on failure This call will return the number of allocated hosts for the specified job. If failures occur and EMsg is provided, this buffer will be populated with a human-readable error message.
MCCJobGetRemainingTime(C,JID,Time,EMsg) C: (void **) address of interface handle
JID: (char *) job id to query (optional)
Time: (long *) output value (remaining time in seconds)
EMsg: (char *) optional error message 		Returns 0 on success or -1 on failure This call will return the remaining wallclock duration/time for the specified job. If failures occur and EMsg is provided, this buffer will be populated with a human-readable error message.

NOTE: This call can be made to cache data for performance reasons by setting the MCCCACHETIME #define to the desired cache duration (in seconds) at build time.
MCCJobListAllocRes(C,JID,OBuf,EMsg) C: (void **) address of interface handle
JID: (char *) job id to query (optional)
OBuf: (char **) output response string (comma delimited host list)
EMsg: (char *) optional error message 		Returns 0 on success or -1 on failure This call will return a list of allocated hosts for the specified job. If failures occur and EMsg is provided, this buffer will be populated with a human-readable error message.
MCCJobMigrate(C,JID,Dest,EMsg) C: (void **) address of interface handle
JID: (char *) job id to migrate (optional)
Dest: (char *) destination cluster or list of hosts (optional)
EMsg: (char *) optional error message 		Returns 0 on success or -1 on failure This call will migrate the specified job. If failures occur and EMsg is provided, this buffer will be populated with a human-readable error message.
MCCJobModify(C,JID,Attr,Val,Op,EMsg) C: (void **) address of interface handle
JID: (char *) id of job to modify (optional - if not specified, modify job in current context)
Attr: (char *) job attribute to modify (see mjobctl)
Val:(char *) value to use when modifying specified attribute
Op:(char *) operation to perform on attribute (one of set, unset, incr, or decr)
EMsg: (char *) optional error message 		Returns 0 on success or -1 on failure This call will modify the attributes of either the specified job (if JID is populated) or the job in the current context (if JID is empty). If failures occur and EMsg is provided, this buffer will be populated with a human-readable error message.
MCCJobQuery(C,JID,OBuf,EMsg) C: (void **) address of interface handle
JID: (char *) job id to query (optional - if not specified, report all jobs)
OBuf: (char **) output response string (reported in XML)
EMsg: (char *) optional error message 		Returns 0 on success or -1 on failure This call will suspend the specified job. If failures occur and EMsg is provided, this buffer will be populated with a human-readable error message.
MCCJobRequeue(C,JID,EMsg) C: (void **) address of interface handle
JID: (char *) job id to requeue (optional)
EMsg: (char *) optional error message 		Returns 0 on success or -1 on failure This call will requeue the specified job. If failures occur and EMsg is provided, this buffer will be populated with a human-readable error message.
MCCJobResume(C,JID,EMsg) C: (void **) address of interface handle
JID: (char *) job id to resume (optional)
EMsg: (char *) optional error message 		Returns 0 on success or -1 on failure This call will resume the specified job. If failures occur and EMsg is provided, this buffer will be populated with a human-readable error message.
MCCJobSignal(C,JID,Signo,EMsg) C: (void **) address of interface handle
JID: (char *) job id to resume (optional)
Signo: (int) job signal to send to job
EMsg: (char *) optional error message 		Returns 0 on success or -1 on failure This call will signal the specified job. If failures occur and EMsg is provided, this buffer will be populated with a human-readable error message.
MCCJobStart(C,JID,EMsg) C: (void **) address of interface handle
JID: (char *) job id to start/run (optional)
EMsg: (char *) optional error message 		Returns 0 on success or -1 on failure This call will immediately run/execute the specified job ignoring policies and dependencies provided that adequate idle resources are available. If failures occur and EMsg is provided, this buffer will be populated with a human-readable error message.
MCCJobSubmit(C,JDesc,JID,EMsg) C: (void **) address of interface handle
JDesc: (char *) XML job description (details.doc, detail.html)
JID: (char **) job id of new job (JID is populated on success only)
EMsg: (char *) optional error message 		Returns 0 on success or -1 on failure This call will submit the specified job. If failures occur and EMsg is provided, this buffer will be populated with a human-readable error message. On success, memory is allocated for JID. JID is then populated with a new job's ID (which can then be used in calls such as MCCJobQuery and MCCJobCancel). The client is responsible for freeing the space allocated for JID.
MCCJobSuspend(C,JID,EMsg) C: (void **) address of interface handle
JID: (char *) job id to suspend (optional)
EMsg: (char *) optional error message 		Returns 0 on success or -1 on failure This call will suspend the specified job. If failures occur and EMsg is provided, this buffer will be populated with a human-readable error message.
MCCNodeModify(C,NID,Attr,Val,Op,EMsg) C: (void **) address of interface handle
NID: (char *) id of node to modify (optional - if not specified, modify job in current context)
Attr: (char *) node attribute to modify (see mnodectl)
Val:(char *) value to use when modifying specified attribute
Op:(char *) operation to perform on attribute (one of set, unset, incr, or decr)
EMsg: (char *) optional error message 		Returns 0 on success or -1 on failure This call will modify the attributes of either the specified node (if NID is populated) or the local node (if NID is empty). If failures occur and EMsg is provided, this buffer will be populated with a human-readable error message.
MCCFree(C) C: (void **) address of interface handle
Returns 0 on success or -1 on failure This call will free allocated interface handle memory

Example 1 - Interfacing to an Explicitly Specified Moab Server

client.c
#include <stdio.h>
#include <stdlib.h>
#include "mapi.h"

int main(

  int   ArgC,
  char *ArgV[],
  char *EnvP[])

  {
  void *C;

  char  EMsg[1024];
  char *OBuf;

  char JobID[32];

  char *JobDescription = "<Job><Arguments>600</Arguments>
<Executable>/bin/sleep</Executable><InitialWorkingDirectory>/home/wightman</InitialWorkingDirectory>\

<NodeCount>1</NodeCount><GroupId>wightman</GroupId><UserId>wightman</UserId>\

<Requested></Requested><Processors>1</Processors></Job>";

  if (MCCInitialize("sn-master03",14522,&C) < 0)
    {
    fprintf(stderr,"cannot initialize interface\n");

    exit(-1);
    }
 
  if (MCCJobSubmit(&C,JobDescription,&OBuf,EMsg) < 0)
    {
    fprintf(stderr,"Error: '%s'\n",
      EMsg);

    exit(-1);
    }

  fprintf(stdout,"Results: Job '%s' successfully submitted\n",
    OBuf);

  snprintf(JobID,sizeof(JobID),"%s",OBuf);

  free(OBuf);

  OBuf = NULL;

  if (MCCJobQuery(&C,JobID,&OBuf,EMsg) < 0)
    {
    fprintf(stderr,"Error: '%s'\n",
      EMsg);

    exit(-1);
    }

  fprintf(stdout,"Results: '%s'\n",
    OBuf);

  free(OBuf);

  OBuf = NULL;

  if (MCCJobCancel(&C,JobID,EMsg) < 0)
    {
    fprintf(stderr,"ERROR: '%s'\n",
      EMsg);

    exit(-1);
    }

  fprintf(stdout,"Job '%s' successfully canceled\n",
    JobID); 

  MCCFree(NULL);

  return(0);
  }  /* END main() */

Example 2 - Interface to Monitor Remaining Time w/in a Batch Job

client.c
#include <stdio.h>
#include <stdlib.h>
#include "mapi.h"

int main(

  int   ArgC,
  char *ArgV[],
  char *EnvP[])

  {
  mulong Time;

  /* from inside a batch job, API will auto allocate the interface and
     discover the appropriate jobid */

  /* monitor time, clean-up and exit if less then 300 seconds remain */

  while (1)
    {
    MCCJobGetRemainingTime(NULL,NULL,&Time,NULL);

    if (Time < 300)
      exit(0);
      
    sleep(1);
    }

  /*NOTREACHED*/

  exit(1);
  }

Building client.c
gcc client.c -I/opt/moab-5.1.0/include/ -L/opt/moab-5.1.0/lib/ -lcmoab -lmoab -lpthread -lm -lmcom -lminit -o client

Low-Level C API

The Moab low-level C API allows direct access to all Moab client services. This API uses the MCCExecute routine and takes as an argument an XML request string. This interface provides a high-speed, low-latency interface directly into the server and provides very broad and flexible cluster, workload, and resource control. This routine creates a direct socket connection to provides the following services:

  • socket creation
  • client authentication
  • server discovery via built-in, configfile, and environment settings
  • command request and response encryption
  • marshalling and de-marshalling of requests and data

Requests are sent using native XML request strings and responses are returned in unformatted native XML. Use of Cluster Resources Consulting Services is recommended for organizations using this interface.

Function Args Return Code Description
MCCInitialize(Host,Port,&C) Host: (char *) optional server hostname or NULL
Port: (int) optional server port or 0
C: (void **) opaque interface handle 		Returns 0 on success or -1 on failure This call initializes the interface between the client and the server. It should be called once before attempting to issue any Moab client calls if connecting to a server at a non-default location. If connecting to the default server, client commands will automatically initialize the interface handle.
MCCExecute(C,RCmd,RBuf,&OBuf,EMsg) C: (void **) address of interface handle
RCmd: (char *) request client command
RBuf: (char *) request command XML
OBuf: (char **) output response string
EMsg: (char *) optional error message 		Returns 0 on success or -1 on failure This call executes the requested command specified by the RBuf string using the client command specified by RCmd. Command output is placed in the OBuf buffer. This buffer is allocated and must be freed after use. The EMsg parameter is a 1024 character array and is optional. If failures occur and EMsg is provided, this buffer will be populated with a human-readable error message.

H.1.2 CLI (Command Line Interface) XML API

All Moab client commands can report results in XML format to allow the information to be easily integrated into peer services, portals, databases, and other applications. To request that a client command report its output in XML, specify the --format=xml flag as in the following example:

showq
> showq --format=xml

<Data>
<Object>queue</Object>
<cluster LocalActiveNodes="1" LocalAllocProcs="1" LocalIdleNodes="0" LocalIdleProcs="3" LocalUpNodes="1" 
  LocalUpProcs="4" RemoteActiveNodes="0" RemoteAllocProcs="0" RemoteIdleNodes="0" RemoteIdleProcs="0" 
  RemoteUpNodes="0" RemoteUpProcs="0" time="1128451812"></cluster>
<queue count="1" option="active">
<job AWDuration="11672" EEDuration="1128451812" Group="[DEFAULT]" JobID="Moab.2" MasterHost="cw2" PAL="2" 
  QOS="bug3" ReqAWDuration="54000" ReqNodes="1" ReqProcs="1" RsvStartTime="1128451812" RunPriority="0" 
  StartPriority="1" StartTime="1128451812" StatPSDed="11886.580000" StatPSUtl="11886.580000" State="Running" 
  SubmissionTime="1128451812" SuspendDuration="0" User="smith"></job>
</queue>
<queue count="1" option="eligible">
<job EEDuration="1128451812" Group="jacksond" JobID="customer.35" QOS="bug" ReqAWDuration="3600" 
  ReqProcs="1" StartPriority="1" StartTime="0" State="Idle" SubmissionTime="1128451812" SuspendDuration="0" 
  User="johnson"></job>
<queue><queue count="0" option="blocked"></queue>
</Data>

Common Query/Control Services

H.2 Resource Management Interfaces

   Moab can monitor, schedule, and control services and resources using multiple protocols.  These protocols include the following:

Using the resource manager interfaces, Moab can do the following:

  • monitor resources (compute host, network, storage, and software license based resources)
    • load configuration, architecture, and feature information
    • load state, utilization, and workload information
    • load policy and ownership information
  • manage resources
    • dynamically reconfigure and reprovision resource hardware (processors, memory, etc.)
    • dynamically reconfigure and reprovision resource software (operating system, application software, filesystem mounts, etc.)
    • dynamically reconfigure and reprovision resource security (VPN's, VLAN's, host security, etc.)
  • monitor workload (batch jobs, interactive jobs, persistent services, dynamic services, distributed services)
    • load state, resource requirement, and required environment information
    • load user, group, and credential information
    • load utilization, resource allocation, and policy information
  • manage workload
    • migrate jobs from one resource to another (intra-cluster and inter-cluster)
    • modify jobs for translation and optimization purposes
    • suspend, resume, checkpoint, restart, and cancel jobs
  • query cluster policies and configuration

H.3 Identity and Credential Management Interfaces

Moab's identity and credential management interfaces allow Moab to exchange credential and user configuration, access, policy, and usage information.

H.4 Accounting Interfaces

Moab accounting interfaces allow Moab to export local utilization statistics, events, and accounting information directly to databases and peer services. Moab interfaces can also import global and remote accounting information for use in local decision making.

H.5 Grid Interfaces

Moab provides interfaces to allow interaction with various grid brokers and services. These interfaces allow Moab to provide services as well as utilize services.

Services Utilized

  • Information Services (import and utilize information service data in making scheduling decisions)
  • Job Migration
  • Data Migration
  • Credential Mapping
  • Security and Delegation

See Moab Workload Manager for Grids for more information on utilized services.

Services Provided

  • Information Services (provide resource, workload, and credential information))
  • Job Migration
  • Data Migration
  • Credential Mapping

See Moab Workload Manager for Grids for more information on provided services.

H.6 Discovery/Directory Services

Moab can import and export key event information regarding workload, cluster resources, cluster and grid services, and other components of hardware and software infrastructure.

By default, these clients communicate with the scheduler using the U.S. Department of Energy (DOE) Scalable Systems Software socket and wire protocols. These protocols are largely HTML- and XML-based, using PKI, 3DES, MD5, challenge, and other security protocols and are documented within the SSS project pages.

As part of this initiative, the scheduler/client protocol has been extended to support multiple socket level protocol standards in communicating with its clients and peer services. These include SingleUseTCP, SSS-HALF, and HTTP. The client socket protocol can be specified by setting the MCSOCKETPROTOCOL parameter to SUTCP, SSS-HALF, or HTTP. Further protocols are being defined and standardized over time and backwards compatibility will be maintained. Documentation on the SSS-HALF implementation can be found within the DOE's SSS Project Notebooks.

H.7 Job Submission and Management Interface

Moab provides interfaces to enable the following services:

  • Resource Availability Query
    • Determine quantity, state, and configuration of configured resources (idle, busy, and down nodes)
    • Determine quantity and configuration of all available resources (idle nodes)
    • Determine resources available subject now and in the future for potential job
    • Determine best target cluster destination for potential job
    • Determine largest/longest job which could start immediately
    • Determine estimated start time for potential job
    • Determine earliest guaranteed start time for potential job
  • Reserve Resources
    • Reserve specific resources for desired time frame
  • Submit Job
    • Submit job to specific cluster
    • Submit job to global job queue
  • Manage Job
    • Hold job
    • Adjust job priority
    • Modify job executable, args, data requirements, job dependencies, duration, hostcount, or other attributes
    • Suspend/resume job
    • Checkpoint/requeue job
    • Cancel job
    • Migrate job
    • Adjust job quality of service (QoS)
  • Query Job
    • Determine job state, utilization, or output results for idle, active, or completed job
    • Determine estimated start time
    • Determine guaranteed start time
© 2001-2008 Cluster Resources, Incorporated