Moab Workload Manager®

13.2 Resource Manager Configuration

• 13.2.1 Defining and Configuring Resource Manager Interfaces
  • 13.2.1.1 Resource Manager Attributes
• 13.2.2 Resource Manager Configuration Details
  • 13.2.2.1 Resource Manager Types
  • 13.2.2.2 Resource Manager Name
  • 13.2.2.3 Resource Manager Location
  • 13.2.2.4 Resource Manager Flags
  • 13.2.2.5 Other Attributes
• 13.2.3 Scheduler/Resource Manager Interactions

13.2.1 Defining and Configuring Resource Manager Interfaces

Moab resource manager interfaces are defined using the RMCFG parameter. This parameter allows specification of key aspects of the interface. In most cases, only the TYPE attribute needs to be specified and Moab determines the needed defaults required to activate and use the selected interface. In the following example, an interface to a Loadleveler resource manager is defined.

RMCFG[orion] TYPE=LL
...

Note that the resource manager is given a label of orion. This label can be any arbitrary site-selected string and is for local usage only. For sites with multiple active resource managers, the labels can be used to distinguish between them for resource manager specific queries and commands.

13.2.1.1 Resource Manager Attributes

The following table lists the possible resource manager attributes that can be configured.

ADMINEXEC
Format:	"jobsubmit"
Default:	NONE
Description:	Normally, when the JOBSUBMITURL is executed, Moab will drop to the UID and GID of the user submitting the job. Specifying an ADMINEXEC of jobsubmit causes Moab to use its own UID and GID instead (usually root). This is useful for some native resource managers where the JOBSUBMITURL is not a user command (such as qsub) but a script that interfaces directly with the resource manager.
Example:	RMCFG[base] ADMINEXEC=jobsubmit Moab will not use the user's UID and GID for executing the JOBSUBMITURL.
AUTHTYPE
Format:	one of CHECKSUM, OTHER, PKI, SECUREPORT, or NONE.
Default:	CHECKSUM
Description:	Specifies the security protocol to be used in scheduler-resource manager communication. Only valid with WIKI based interfaces.
Example:	RMCFG[base] AUTHTYPE=CHECKSUM Moab requires a secret key based checksum associated with each resource manager message.
BANDWIDTH
Format:	<FLOAT>[{M|G|T}]
Default:	-1 (unlimited)
Description:	Specifies the maximum deliverable bandwidth between the Moab server and the resource manager for staging jobs and data. Bandwidth is specified in units per second and defaults to a unit of MB/s. If a unit modifier is specified, the value is interpreted accordingly (M - megabytes/sec, G - gigabytes/sec, T - terabytes/sec).
Example:	RMCFG[base] BANDWIDTH=340G Moab will reserve up to 340 GB of network bandwidth when scheduling job and data staging operations to and from this resource manager.
CHECKPOINTSIG
Format:	one of suspend, <INTEGER>, or SIG<X>
Default:	---
Description:	Specifies what signal to send the resource manager when a job is checkpointed. (See Checkpoint Overview.)
Example:	RMCFG[base] CHECKPOINTSIG=SIGKILL Moab routes the signal SIGKILL through the resource manager to the job when a job is checkpointed.
CHECKPOINTTIMEOUT
Format:	[[[DD:]HH:]MM:]SS
Default:	0 (no timeout)
Description:	Specifies how long Moab waits for a job to checkpoint before canceling it. If set to 0, Moab does not cancel the job if it fails to checkpoint. (See Checkpoint Overview.)
Example:	RMCFG[base] CHECKPOINTTIMEOUT=5:00 Moab cancels any job that has not exited 5 minutes after receiving a checkpoint request.
CLIENT
Format:	<PEER>
Default:	use name of resource manager for peer client lookup
Description:	If specified, the resource manager will use the peer value to authenticate remote connections. (See configuring peers). If not specified, the resource manager will search for a CLIENTCFG entry of RM:<RMNAME> in the moab-private.cfg file.
Example:	RMCFG[clusterBI] CLIENT=clusterB Moab will look up and use information for peer clusterB when authenticating the clusterBI resource manager.
CLUSTERQUERYURL
Format:	[file://<path> | http://<address> | <path>]   If file:// is specified, Moab treats the destination as a flat text file; if http:// is specified, Moab treats the destination as a hypertext transfer protocol file; if just a path is specified, Moab treats the destination as an executable.
Default:	---
Description:	Specifies how Moab queries the resource manager. (See Native RM, URL Notes, and interface details.)
Example:	RMCFG[base] CLUSTERQUERYURL=file:///tmp/cluster.config Moab reads /tmp/cluster.config when it queries base resource manager.
CONFIGFILE
Format:	<STRING>
Default:	N/A
Description:	Specifies the resource manager specific configuration file that must be used to enable correct API communication. Only valid with LL- and SLURM-based interfaces.
Example:	RMCFG[base] TYPE=LL CONFIGFILE=/home/loadl/loadl_config The scheduler uses the specified file when establishing the resource manager/scheduler interface connection.
DATARM
Format:	<RM NAME>
Default:	N/A
Description:	If specified, the resource manager uses the given storage resource manager to handle staging data in and out.
Example:	RMCFG[clusterB] DATARM=clusterB_storage When data staging is required by jobs starting/completing on clusterB, Moab uses the storage interface defined by clusterB_storage to stage and monitor the data.
DEFAULTCLASS
Format:	<STRING>
Default:	N/A
Description:	Specifies the class to use if jobs submitted via this resource manager interface do not have an associated class.
Example:	RMCFG[internal] DEFAULTCLASS=batch Moab assigns the class batch to all jobs from the resource manager internal that do not have a class assigned. If you are using PBS as the resource manager, a job will never come from PBS without a class, and the default will never apply.
DEFAULT.JOB
Format:	<STRING>
Default:	N/A
Description:	Specifies the job template to use to set various job attributes that are not specified by the submitter.
Example:	RMCFG[base] DEFAULT.JOB=defjob Moab uses the defjob job template to identify and apply job attribute defaults.
DEFAULTHIGHSPEEDADAPTER
Format:	<STRING>
Default:	sn0
Description:	Specifies the default high speed switch adapter to use when starting LoadLeveler jobs (supported in version 4.2.2 and higher of Moab and 3.2 of LoadLeveler).
Example:	RMCFG[base] DEFAULTHIGHSPEEDADAPTER=sn1 The scheduler will start jobs requesting a high speed adapter on sn1.
DESCRIPTION
Format:	<STRING>
Default:	N/A
Description:	Specifies the human-readable description for the resource manager interface. If white space is used, the description should be quoted.
Example:	RMCFG[ganglia] TYPE=NATIVE:ganglia DESCRIPTION='resource monitor providing extended resource utilization stats' Moab annotates the ganglia resource manager accordingly.
ENV
Format:	Semi-colon delimited (;) list of <KEY>=<VALUE> pairs
Default:	MOABHOMEDIR=<MOABHOMEDIR>
Description:	Specifies a list of environment variables that will be passed to URL's of type 'exec://' for that resource manager.
Example:	RMCFG[base] ENV=HOST=node001;RETRYTIME=50 RMCFG[base] CLUSTERQUERYURL=exec:///opt/moab/tools/cluster.query.pl RMCFG[base] WORKLOADQUERYURL=exec:///opt/moab/tools/workload.query.pl The environment variables HOST and RETRYTIME (with values 'node001' and '50' respectively) are passed to the /opt/moab/tools/cluster.query.pl and /opt/moab/tools/workload.query.pl when they are executed.
EPORT
Format:	<INTEGER>
Default:	N/A
Description:	Specifies the event port to use to receive resource manager based scheduling events.
Example:	RMCFG[base] EPORT=15017 The scheduler will look for scheduling events from the resource manager host at port 15017.
FAILTIME
Format:	<[[[DD:]HH:]MM:]SS
Default:	N/A
Description:	Specifies how long a resource manager must be down before any failure triggers associated with the resource manager fire.
Example:	RMCFG[base] FAILTIME=3:00 If the base resource manager is down for three minutes, any resource manager failure triggers fire.
FLAGS
Format:	comma delimited list of zero or more of the following: asyncstart, autostart, autosync, client, fullcp, executionServer, grid, hostingCenter, ignqueuestate, private, pushslavejobupdates, report, shared, slavepeer or static
Default:	N/A
Description:	Specifies various attributes of the resource manager. See Flag Details for more information.
Example:	RMCFG[base] FLAGS=static,slavepeer Moab uses this resource manager to perform a single update of node and job objects reported elsewhere.
FLOWINTERVAL
Format:	[[[DD:]HH:]MM:]SS
Default:	01:00:00 (one hour)
Description:	Specifies the duration of the flow control sliding window.
Example:	RMCFG[base] FLOWINTERVAL=00:30:00 FLOWMETRIC=jobs FLOWLIMIT=30 The scheduler limits jobs running on this resource manager to no more than 30 jobs every 30 minutes.
FLOWLIMIT
Format:	<INTEGER>
Default:	1
Description:	Specifies the limit of flow metric consumption allowed within the sliding window.
Example:	RMCFG[base] FLOWINTERVAL=00:30:00 FLOWMETRIC=jobs FLOWLIMIT=30 The scheduler limits jobs running on this resource manager to no more than 30 jobs every 30 minutes.
FLOWMETRIC
Format:	one of jobs, procs, nodes, procseconds, or peseconds
Default:	jobs
Description:	Specifies the metric of consumption of the flow control sliding window.
Example:	RMCFG[base] FLOWINTERVAL=00:30:00 FLOWMETRIC=jobs FLOWLIMIT=30 The scheduler limits jobs running on this resource manager to no more than 30 jobs every 30 minutes.
FNLIST
Format:	comma delimited list of zero or more of the following: clusterquery, jobcancel, jobrequeue, jobresume, jobstart, jobsuspend, queuequery, resourcequery or workloadquery
Default:	N/A
Description:	By default, a resource manager utilizes all functions supported to query and control batch objects. If this parameter is specified, only the listed functions are used.
Example:	RMCFG[base] FNLIST=queuequery Moab only uses this resource manager interface to load queue configuration information.
HOST
Format:	<STRING>
Default:	localhost
Description:	The host name of the machine on which the resource manager server is running.
Example:	RMCFG[base] host=server1
IGNHNODES
Format:	<BOOLEAN>
Default:	FALSE
Description:	Specifies whether to read in the PBSPro host nodes. This parameter is used in conjunction with USEVNODES. When both are set to TRUE, the host nodes are not queried.
Example:	RMCFG[pbs] IGNHNODES=TRUE
JOBCANCELURL
Format:	<protocol>://[<host>[:<port>]][<path>]
Default:	---
Description:	Specifies how Moab cancels jobs via the resource manager. (See URL Notes below.)
Example:	RMCFG[base] JOBCANCELURL=exec:///opt/moab/job.cancel.lsf.pl Moab executes /opt/moab/job.cancel.lsf.pl to cancel specific jobs.
JOBEXTENDDURATION
Format:	[[[DD:]HH:]MM:]SS[,[[[DD:]HH:]MM:]SS][!][<] (or <MIN TIME>[,<MAX TIME>][!])
Default:	---
Description:	Specifies the minimum and maximum amount of time that can be added to a job's walltime if it is possible for the job to be extended. (See MINWCLIMIT.) As the job runs longer than its current specified minimum wallclock limit (-l minwclimit, for example), Moab attempts to extend the job's limit by the minimum JOBEXTENDDURATION. This continues until either the extension can no longer occur (it is blocked by a reservation or job), the maximum JOBEXTENDDURATION is reached, or the user's specified wallclock limit (-l wallclock) is reached. When a job is extended, it is marked as PREEMPTIBLE, unless the '!' is appended to the end of the configuration string. If the '<' is at the end of the string, however, the job is extended the maximum amount possible.
Example:	RMCFG[base] JOBEXTENDDURATION=30,1:00:00 Moab extends a job's walltime by 30 seconds each time the job is about to run out of walltime until it is bound by one hour, a reservation/job, or the job's original "maximum" wallclock limit.
JOBMODIFYURL
Format:	<protocol>://[<host>[:<port>]][<path>]
Default:	---
Description:	Specifies how Moab modifies jobs via the resource manager. (See Dynamic Jobs, URL Notes, and interface details.)
Example:	RMCFG[base] JOBMODIFYURL=exec://$TOOLSDIR/job.modify.dyn.pl Moab executes /opt/moab/job.modify.dyn.pl to modify specific jobs.
JOBPREEMPTURL
Format:	><protocol>://[<host>[:<port>]][<path>]
Default:	---
Description:	Specifies how Moab preempts jobs via the resource manager. (See URL Notes below.)
Example:	RMCFG[base] JOBPREEMPTURL=exec:///opt/moab/job.preempt.condor.pl Moab executes /opt/moab/job.preempt.condor.pl to preempt specific jobs.
JOBRSVRECREATE
Format:	Boolean
Default:	TRUE
Description:	Specifies whether Moab will re-create a job reservation each time job information is updated by a resource manager. (See Considerations for Large Clusters for more information.)
Example:	RMCFG[base] JOBRSVRECREATE=FALSE Moab only creates a job reservation once when the job first starts.
JOBSTARTURL
Format:	<protocol>://[<host>[:<port>]][<path>]
Default:	---
Description:	Specifies how Moab starts jobs via the resource manager. (See URL Notes below.)
Example:	RMCFG[base] JOBSTARTURL=http://orion.bsu.edu:1322/moab/jobstart.cgi Moab triggers the jobstart.cgi script via http to start specific jobs.
JOBSUBMITURL
Format:	<protocol>://[<host>[:<port>]][<path>]
Default:	---
Description:	Specifies how Moab submits jobs to the resource manager. (See URL Notes below.)
Example:	RMCFG[base] JOBSUBMITURL=sql://dbserver.flc.com:3554/admin Moab submits jobs directly to the database located on host dbserver.flc.com.
JOBVALIDATEURL
Format:	<protocol>://[<host>[:<port>]][<path>]
Default:	---
Description:	Specifies how Moab validates newly submitted jobs. (See URL Notes below.) If the script returns with a non-zero exit code, the job is rejected. (See User Proxying/Alternate Credentials.)
Example:	RMCFG[base] JOBVALIDATEURL=exec://$TOOLS/job.validate.pl Moab executes the 'job.validate.pl' script when jobs are submitted to verify they are acceptable.
MAXDSOP
Format:	<INTEGER>
Default:	-1 (unlimited)
Description:	Specifies the maximum number of data staging operations that may be simultaneously active.
Example:	RMCFG[ds] MAXDSOP=16
MAX.JOB
Format:	<STRING>
Default:	---
Description:	Specifies the job template to use to check various maximum/excluded job attributes that are specified by the submitter.
Example:	RMCFG[base] MAX.JOB=maxjob Moab will use the maxjob job template to identify and enforce maximum/excluded job attributes.
MAXJOBPERMINUTE
Format:	<INTEGER>
Default:	-1 (unlimited)
Description:	Specifies the maximum number of jobs allowed to start per minute via the resource manager.
Example:	RMCFG[base] MAXJOBPERMINUTE=5 The scheduler only allows five jobs per minute to launch via the resource manager base.
MAXJOBS
Format:	<INTEGER>
Default:	0 (limited only by the Moab MAXJOB setting)
Description:	Specifies the maximum number of active jobs that this interface is allowed to load from the resource manager. Only works with Moab peer resource managers at this time.
Example:	RMCFG[cluster1] SERVER=moab://cluster1 MAXJOBS=200 The scheduler loads up to 200 active jobs from the remote Moab peer cluster1.
MINETIME
Format:	<INTEGER>
Default:	1
Description:	Specifies the minimum time in seconds between processing subsequent scheduling events.
Example:	RMCFG[base] MINETIME=5 The scheduler batch-processes scheduling events that occur less than five seconds apart.
MIN.JOB
Format:	<STRING>
Default:	---
Description:	Specifies the job template to use to check various minimum/required job attributes that are specified by the submitter.
Example:	RMCFG[base] MIN.JOB=minjob Moab uses the minjob job template to identify and enforce minimum/required job attributes.
NMPORT
Format:	<INTEGER>
Default:	(any valid port number)
Description:	Allows specification of the resource manager's node manager port and is only required when this port has been set to a non-default value.
Example:	RMCFG[base] NMPORT=13001 The scheduler contacts the node manager located on each compute node at port 13001.
NODEFAILURERSVPROFILE
Format:	<STRING>
Default:	N/A
Description:	Specifies the rsv template to use when placing a reservation onto failed nodes. (See also NODEFAILURERESERVETIME.)
Example:	moab.cfg RMCFG[base] NODEFAILURERSVPROFILE=long RSVPROFILE[long] DURATION=25:00 RSVPROFILE[long] USERLIST=john The scheduler will use the long rsv profile when creating reservations over failed nodes belonging to base.
NODESTATEPOLICY
Format:	one of OPTIMISTIC or PESSIMISTIC
Default:	PESSIMISTIC
Description:	Specifies how Moab should determine the state of a node when multiple resource managers are reporting state. OPTIMISTIC specifies that if any resource manager reports a state of up, that state will be used. PESSIMISTIC specifies that if any resource manager reports a state of down, that state will be used.
Example:	moab.cfg RMCFG[ganglia] TYPE=NATIVE CLUSTERQUERYURL=ganglia:// RMCFG[ganglia] FLAGS=SLAVEPEER NODESTATEPOLICY=OPTIMISTIC
OMAP
Format:	<protocol>://[<host>[:<port>]][<path>]
Default:	---
Description:	Specifies an object map file that is used to map credentials and other objects when using this resource manager peer. (See Grid Credential Management for full details.)
Example:	moab.cfg RMCFG[peer1] OMAP=file:///opt/moab/omap.dat When communicating with the resource manager peer1, objects are mapped according to the rules defined in the /opt/moab/omap.dat file.
POLLINTERVAL
Format:	[[[DD:]HH:]MM:]SS
Default:	30
Description:	Specifies how often the scheduler will poll the resource manager for information.
Example:	RMCFG[base] POLLINTERVAL=1:00 Moab contacts resource manager base every minute for updates.
POLLTIMEISRIGID
Format:	<BOOLEAN>
Default:	FALSE
Description:	Determines whether the POLLINTERVAL parameter is interpreted as an interval or a set time for contacting.
Example:	RMCFG[base] POLLTIMEISRIGID=TRUE POLLINTERVAL=1:00:00 Moab polls the resource manager at startup and on the hour.
PORT
Format:	<INTEGER>
Default:	0
Description:	Specifies the port on which the scheduler should contact the associated resource manager. The value '0' specifies that the resource manager default port should be used.
Example:	RMCFG[base] TYPE=PBS HOST=cws PORT=20001 Moab attempts to contact the PBS server daemon on host cws, port 20001.
PTYSTRING
Format:	<STRING>
Default:	srun -n1 -N1 --pty
Description:	When a SLURM interactive job is submitted, it builds an salloc command that gets the requested resources and an srun command that creates a terminal session on one of the nodes. The srun command is called the PTYString. PTYString is configured in moab.cfg. There are two special things you can do with PTYString: You can have PTYSTRING=$salloc which says to use the default salloc command (SallocDefaultCommand, look in the slurm.conf man page) defined in slurm.conf. Internally, Moab won't add a PTYString because SLURM will call the SallocDefaultCommand. As in the example below, you can add $SHELL. $SHELL will be expanded to either what you request on the command line (such as msub -S /bin/tcsh -l) or to the value of $SHELL in your current session. PTYString works only with SLURM.
Example:	RMCFG[slurm] PTYSTRING="srun -n1 -N1 --pty --preserve-env $SHELL"
RESOURCETYPE
Format:	{COMPUTE|FS|LICENSE|NETWORK}
Default:	---
Description:	Specifies which type of resource this resource manager is configured to control. See Native Resource Managers for more information.
Example:	RMCFG[base] TYPE=NATIVE RESOURCETYPE=FS Resource manager base will function as a NATIVE resource manager and control file systems.
RMSTARTURL
Format:	[exec://<path> | http://<address> | <path>]   If exec:// is specified, Moab treats the destination as an executable file; if http:// is specified, Moab treats the destination as a hypertext transfer protocol file.
Default:	---
Description:	Specifies how Moab starts the resource manager.
Example:	RMCFG[base] RMSTARTURL=exec:///tmp/nat.start.pl Moab executes /tmp/nat.start.pl to start the resource manager base.
RMSTOPURL
Format:	[exec://<path> | http://<address> | <path>]   If exec:// is specified, Moab treats the destination as an executable file; if http:// is specified, Moab treats the destination as a hypertext transfer protocol file.
Default:	---
Description:	Specifies how Moab stops the resource manager.
Example:	RMCFG[base] RMSTOPURL=exec:///tmp/nat.stop.pl Moab executes /tmp/nat.stop.pl to stop the resource manager base.
SBINDIR
Format:	<PATH>
Default:	N/A
Description:	For use with TORQUE; specifies the location of the TORQUE system binaries (supported in TORQUE 1.2.0p4 and higher).
Example:	RMCFG[base] TYPE=pbs SBINDIR=/usr/local/torque/sbin Moab tells TORQUE that its system binaries are located in /usr/local/torque/sbin.
SERVER
Format:	<URL>
Default:	N/A
Description:	Specifies the resource management service to use. If not specified, the scheduler locates the resource manager via built-in defaults or, if available, with an information service.
Example:	RMCFG[base] server=ll://supercluster.org:9705 Moab attempts to use the Loadleveler scheduling API at the specified location.
SET.JOB
Format:	<STRING>
Default:	N/A
Description:	Specifies the job template to use to force various job attributes regardless of whether or not they are specified by the submitter.
Example:	JOBCFG[setjob] QOS=fast RMCFG[base] SET.JOB=setjob Moab uses the setjob job template to identify and enforce mandatory job attributes.
SLURMFLAGS
Format:	<STRING>
Default:	N/A
Description:	Specifies characteristics of the SLURM resource manager interface.
Example:	RMCFG[slurm] SLURMFLAGS=COMPRESSOUTPUT Moab uses the specified flag to determine interface characteristics with SLURM. The COMPRESSOUTPUT flag instructs Moab to use the compact hostlist format for job submissions to SLURM. The flag NODEDELTAQUERY instructs Moab to request delta node updates when it queries SLURM for node configuration.
SOFTTERMSIG
Format:	<INTEGER> or SIG<X>
Default:	---
Description:	Specifies what signal to send the resource manager when a job reaches its soft wallclock limit. (See JOBMAXOVERRUN.)
Example:	RMCFG[base] SOFTTERMSIG=SIGUSR1 Moab routes the signal SIGUSR1 through the resource manager to the job when a job reaches its soft wallclock limit.
STAGETHRESHOLD
Format:	[[[DD:]HH:]MM:]SS
Default:	N/A
Description:	Specifies the maximum time a job waits to start locally before considering being migrated to a remote peer. In other words, if a job's start time on a remote cluster is less than the start time on the local cluster, but the difference between the two is less than STAGETHRESHOLD, then the job is scheduled locally. The aim is to avoid job/data staging overhead if the difference in start times is minimal. If this attribute is used, backfill is disabled for the associated resource manager.
Example:	RMCFG[remote_cluster] STAGETHRESHOLD=00:05:00 Moab only migrates jobs to remote_cluster if the jobs can start five minutes sooner on the remote cluster than they could on the local cluster.
STARTCMD
Format:	<STRING>
Default:	N/A
Description:	Specifies the full path to the resource manager job start client. If the resource manager API fails, Moab executes the specified start command in a second attempt to start the job. Moab calls the start command with the format '<CMD> <JOBID> -H <HOSTLIST>' unless the environment variable 'MOABNOHOSTLIST' is set in which case Moab will only pass the job ID.
Example:	RMCFG[base] STARTCMD=/usr/local/bin/qrun Moab uses the specified start command if API failures occur when launching jobs.
SUBMITCMD
Format:	<STRING>
Default:	N/A
Description:	Specifies the full path to the resource manager job submission client.
Example:	RMCFG[base] SUBMITCMD=/usr/local/bin/qsub Moab uses the specified submit command when migrating jobs.
SUBMITPOLICY
Format:	one of NODECENTRIC or PROCCENTRIC
Default:	PROCCENTRIC
Description:	If set to NODECENTRIC, each specified node requested by the job is interpreted as a true compute host, not as a task or processor.
Example:	RMCFG[base] SUBMITPOLICY=NODECENTRIC Moab uses the specified submit policy when migrating jobs.
SUSPENDSIG
Format:	<INTEGER> (valid UNIX signal between 1 and 64)
Default:	--- (resource manager specific default)
Description:	If set, Moab sends the specified signal to a job when a job suspend request is issued.
Example:	RMCFG[base] SUSPENDSIG=19 Moab uses the specified suspend signal when suspending jobs within the base resource manager. SUSPENDSIG should not be used with TORQUE or other PBS-based resource managers.
SYNCJOBID
Format:	<BOOLEAN>
Default:	---
Description:	Specifies that Moab should migrate jobs to the local resource manager queue with a job ID matching the job's Moab-assigned job ID (only available with SLURM.)
Example:	RMCFG[base] TYPE=wiki:slurm SYNCJOBID=TRUE Moab migrates jobs to the SLURM queue with a jobid matching the Moab-assigned job ID.
SYSTEMMODIFYURL
Format:	[exec://<path> | http://<address> | <path>]   If exec:// is specified, Moab treats the destination as an executable file; if http:// is specified, Moab treats the destination as a hypertext transfer protocol file.
Default:	---
Description:	Specifies how Moab modifies attributes of the system. This interface is used in data staging.
Example:	RMCFG[base] SYSTEMMODIFYURL=exec:///tmp/system.modify.pl Moab executes /tmp/system.modify.pl when it modifies system attributes in conjunction with the resource manager base.
SYSTEMQUERYURL
Format:	[file://<path> | http://<address> | <path>]   If file:// is specified, Moab treats the destination as a flat text file; if http:// is specified, Moab treats the destination as a hypertext transfer protocol file; if just a path is specified, Moab treats the destination as an executable.
Default:	---
Description:	Specifies how Moab queries attributes of the system. This interface is used in data staging.
Example:	RMCFG[base] SYSTEMQUERYURL=file:///tmp/system.query Moab reads /tmp/system.query when it queries the system in conjunction with base resource manager.
TARGETUSAGE
Format:	<INTEGER>[%]
Default:	90%
Description:	Amount of resource manager resources to explicitly use. In the case of a storage resource manager, indicates the target usage of data storage resources to dedicate to active data migration requests. If the specified value contains a percent sign (%), the target value is a percent of the configured value. Otherwise, the target value is considered to be an absolute value measured in megabytes (MB).
Example:	RMCFG[storage] TYPE=NATIVE RESOURCETYPE=storage RMCFG[storage] TARGETUSAGE=80% Moab schedules data migration requests to never exceed 80% usage of the storage resource manager's disk cache and network resources.
TIMEOUT
Format:	<INTEGER>
Default:	30
Description:	Time (in seconds) the scheduler waits for a response from the resource manager.
Example:	RMCFG[base] TIMEOUT=40 Moab waits 40 seconds to receive a response from the resource manager before timing out and giving up. Moab tries again on the next iteration.
TRANSLATIONSCRIPT
Format:	<STRING>
Default:	---
Description:	Script to be inserted into user command file if migration involves translation from one resource manager type to another. The script takes two arguments where the first is the source resource manager type and the second is the destination resource manager type. Types can be any of the following: PBS, SLURM, LSF, SGE, LOADLEVELER, or BPROC.
Example:	RMCFG[base] TRANSLATIONSCRIPT=/opt/moab/tools/tscript.sh Moab inserts a line that will source the specified script into the start of each translated job command file.
TRIGGER
Format:	<TRIG_SPEC>
Default:	---
Description:	A trigger specification indicating behaviors to enforce in the event of certain events associated with the resource manager, including resource manager start, stop, and failure.
Example:	RMCFG[base] TRIGGER=<X>
TYPE
Format:	<RMTYPE>[:<RMSUBTYPE>] where <RMTYPE> is one of the following: TORQUE, LL, LSF, NATIVE, PBS, RMS, SGE, SSS, or WIKI and the optional <RMSUBTYPE> value is one of RMS.
Default:	PBS
Description:	Specifies type of resource manager to be contacted by the scheduler. For TYPE WIKI, AUTHTYPE must be set to CHECKSUM. The <RMSUBTYPE> option is currently only used to support Compaq's RMS resource manager in conjunction with PBS. In this case, the value PBS:RMS should be specified.
Example:	RMCFG[clusterA] TYPE=PBS HOST=clusterA PORT=15003 RMCFG[clusterB] TYPE=PBS HOST=clusterB PORT=15005 Moab interfaces to two different PBS resource managers, one located on server clusterA at port 15003 and one located on server clusterB at port 15005.
UCALLOCDURATION
Format:	[[[DD:]HH:]MM:SS
Default:	--- (no allocation duration)
Description:	Specifies the minimum duration utility computing resources are allocated when the resource manager's UCTHRESHOLD is satisfied. This feature is used when allocating automatic utility computing resources.
Example:	RMCFG[base] UCTHRESHOLD=00:20:00 RMCFG[base] UCALLOCDURATION=00:30:00 RMCFG[base] UCALLOCSIZE=2 The resource manager allocates two processors at a time for a period of at least 30 minutes if the cluster maintains a 20-minute backlog for more than three minutes.
UCALLOCSIZE
Format:	<INTEGER> (processors)
Default:	1
Description:	Specifies the number of additional nodes to allocate each time a dynamic utility computing threshold is reached. This feature is used primarily with utility computing resources. Either UCMAXSIMZE or UCALLOCSIZE must be specified to enable performance or threshold based automatic utility computing usage.
Example:	RMCFG[base] UCTHRESHOLD=00:20:00 RMCFG[base] UCALLOCSIZE=4 The resource manager allocates four additional nodes each time the utility computing threshold is reached.
UCMAXSIZE
Format:	<INTEGER> (processors)
Default:	1
Description:	Specifies the maximum number of nodes the local cluster can allocate in response to utility computing thresholds. This feature is used primarily with utility computing resources. Either UCMAXSIMZE or UCALLOCSIZE must be specified to enable performance or threshold based automatic utility computing usage.
Example:	RMCFG[base] UCMAXSIZE=256 Moab may not allocate more than a total of 256 processors from the utility computing resource even if the utility computing threshold is in violation.
UCTHRESHOLD
Format:	[[[DD:]HH:]MM:SS
Default:	--- (no activation threshold)
Description:	Specifies the cluster backlog duration required before the resource manager automatically activates. This feature is used primarily to activate utility computing resources. This parameter is required to enable performance or threshold based automatic utility computing usage.
Example:	RMCFG[base] UCTHRESHOLD=00:20:00 The resource manager should be activated if the cluster obtains a 20 minute backlog.
UCTHRESHOLDDURATION
Format:	[[[DD:]HH:]MM:SS
Default:	--- (no threshold duration)
Description:	Specifies how long the resource manager's UCTHRESHOLD must be satisfied before resource manager activation is allowed. This parameter prevents statistical spikes from causing unnecessary utility computing allocations. This feature is used primarily to activate utility computing resources.
Example:	RMCFG[base] UCTHRESHOLD=00:20:00 RMCFG[base] UCTHRESHOLDDURATION=00:03:00 Utility computing resources should be allocated if the cluster maintains a 20 minute backlog for more than three minutes.
USEVNODES
Format:	<BOOLEAN>
Default:	FALSE
Description:	Specifies whether to schedule on PBS virtual nodes. When set to TRUE, Moab queries PBSPro for vnodes and puts jobs on vnodes rather than "hosts." In some systems, such as PBS + Altix, it may not be desirable to read in the host nodes; for such situations refer to the IGNHNODES parameter.
Example:	RMCFG[pbs] USEVNODES=TRUE
VARIABLES
Format:	<VAR>=<VAL>[,VAR>=<VAL>]
Default:	---
Description:	Opaque resource manager variables.
Example:	RMCFG[base] VARIABLES=SCHEDDHOST=head1 Moab associates the variable SCHEDDHOST with the value head1 on resource manager base.
VERSION
Format:	<STRING>
Default:	SLURM: 10200 (i.e., 1.2.0)
Description:	Resource manager-specific version string.
Example:	RMCFG[base] VERSION=10124 Moab assumes that resource manager base has a version number of 1.1.24.
VMOWNERRM
Format:	<STRING>
Default:	---
Description:	Used with provisioning resource managers that can create VMs. It specifies the resource manager that will own any VMs created by the resource manager.
Example:	RMCFG[torque] RMCFG[prov] RESOURCETYPE=PROV VMOWNERRM=torque
WORKLOADQUERYURL
Format:	[file://<path> | http://<address> | <path>]   If file:// is specified, Moab treats the destination as a flat text file; if http:// is specified, Moab treats the destination as a hypertext transfer protocol file; if just a path is specified, Moab treats the destination as an executable.
Default:	---
Description:	Specifies how Moab queries the resource manager for workload information. (See Native RM, Dynamic Jobs, URL Notes, and interface details.)
Example:	RMCFG[dynamic_jobs] WORKLOADQUERYURL=exec://$TOOLSDIR/job.query.dyn.pl Moab executes /opt/moab/tools/job.query.dyn.pl to obtain updated workload information from resource manager dynamic_jobs.

URL notes

URL parameters can load files by using the file, exec, http, and sql protocols.

For the protocol file, Moab loads the data directly from the text file pointed to by path.

RMCFG[base] SYSTEMQUERYURL=file:///tmp/system.query

For the protocol exec, Moab executes the file pointed to by path and loads the output written to STDOUT. If the script requires arguments, you can use a question mark (?) between the script name and the arguments, and an ampersand (&) for each space.

RMCFG[base] JOBVALIDATEURL=exec://$TOOLS/job.validate.pl
RMCFG[native] CLUSTERQUERYURL=exec://opt/moab/tools/cluster.query.pl?-group=group1&-arch=x86

For the protocol http, Moab loads the data from the web based HTTP (hypertext transfer protocol) destination.

RMCFG[base] JOBSTARTURL=http://orion.bsu.edu:1322/moab/jobstart.cgi

For the protocol sql, Moab loads the data from the specified database.

RMCFG[base] JOBSUBMITURL=sql://dbserver.flc.com:3554/admin

13.2.2 Resource Manager Configuration Details

As with all scheduler parameters, RMCFG follows the syntax described within the Parameters Overview.

13.2.2.1 Resource Manager Types

The RMCFG parameter allows the scheduler to interface to multiple types of resource managers using the TYPE or SERVER attributes. Specifying these attributes, any of the following listed resource managers may be supported. To further assist in configuration, Integration Guides are provided for PBS, SGE, and Loadleveler systems.

TYPE	Resource Managers	Details
LL	Loadleveler version 2.x and 3.x	N/A
LSF	Platform's Load Sharing Facility, version 5.1 and higher	N/A
Moab	Moab Workload Manager	Use the Moab peer-to-peer (grid) capabilities to enable grids and other configurations. (See Grid Configuration.)
Native	Moab Native Interface	Used for connecting directly to scripts, files, databases, and web services. (See Managing Resources Directly with the Native Interface.)
PBS	TORQUE (all versions), OpenPBS (all versions), PBSPro (all versions)	N/A
RMS	RMS (for Quadrics based systems)	For development use only; not production quality.
SGE	Sun Grid Engine version 5.3 and higher	N/A
SSS	Scalable Systems Software Project version 2.0 and higher	N/A
WIKI	Wiki interface specification version 1.0 and higher	Used for LRM, YRM, ClubMASK, BProc, SLURM, and others.

13.2.2.2 Resource Manager Name

Moab can support more than one resource manager simultaneously. Consequently, the RMCFG parameter takes an index value such as RMCFG[clusterA]. This index value essentially names the resource manager (as done by the deprecated parameter RMNAME). The resource manager name is used by the scheduler in diagnostic displays, logging, and in reporting resource consumption to the allocation manager. For most environments, the selection of the resource manager name can be arbitrary.

13.2.2.3 Resource Manager Location

The HOST, PORT, and SERVER attributes can be used to specify how the resource manager should be contacted. For many resource managers (such as OpenPBS, PBSPro, Loadleveler, SGE, and LSF) the interface correctly establishes contact using default values. These parameters need only to be specified for resource managers such as the WIKI interface (that do not include defaults) or with resources managers that can be configured to run at non-standard locations (such as PBS). In all other cases, the resource manager is automatically located.

13.2.2.4 Resource Manager Flags

The FLAGS attribute can be used to modify many aspects of a resources manager's behavior.

Flag	Description
ASYNCSTART	Jobs started on this resource manager start asynchronously. In this case, the scheduler does not wait for confirmation that the job correctly starts before proceeding. (See Large Cluster Tuning for more information.)
AUTOSTART	Jobs staged to this resource manager do not need to be explicitly started by the scheduler. The resource manager itself handles job launch.
AUTOSYNC	Resource manager starts and stops together with Moab. This requires that the resource manager support a resource manager start and stop API or the RMSTARTURL and RMSTOPURL attributes are set.
BECOMEMASTER	Nodes reported by this resource manager will transfer ownership to this resource manager if they are currently owned by another resource manager that does not have this flag set.
CLIENT	A client resource manager object is created for diagnostic/statistical purposes or to configure Moab's interaction with this resource manager. It represents an external entity that consumes server resources or services, allows a local administrator to track this usage, and configures specific policies related to that resource manager. A client resource manager object loads no data and provides no services.
CLOCKSKEWCHECKING	Setting CLOCKSKEWCHECKING allows you to configure clock skew adjustments. Most of the time it is sufficient to use an NTP server to keep the clocks in your system synchronized.
COLLAPSEDVIEW	The resource manager masks details about local workload and resources and presents only information relevant to the remote server.
DYNAMICCRED	The resource manager creates credentials within the cluster as needed to support workload. (See Identity Manager Overview.)
EXECUTIONSERVER	The resource manager is capable of launching and executing batch workload.
FSISREMOTE	Add this flag if the working file system doesn't exist on the server to prevent Moab from validating files and directories at migration.
FULLCP	Always checkpoint full job information (useful with Native resource managers).
HOSTINGCENTER	The resource manager interface is used to negotiate an adjustment in dynamic resource access.
IGNQUEUESTATE	The queue state reported by the resource manager should be ignored. May be used if queues must be disabled inside of a particular resource manager to allow an external scheduler to properly operate.
LOCALRSVEXPORT	When using Moab-type resource managers, Moab will export local reservations when requested.
MIGRATEALLJOBATTRIBUTES	When set, this flag causes additional job information to be migrated to the resource manager; additional job information includes things such as node features applied via CLASSCFG[name] DEFAULT.FEATURES, the account to which the job was submitted, and job walltime limit.
NOAUTORES	If the resource manager does not report CPU usage to Moab because CPU usage is at 0%, Moab assumes full CPU usage. When set, Moab recognizes the resource manager report as 0% usage. This is only valid for PBS.
PRIVATE	The resources and workload reported by the resource manager are not reported to non-administrator users.
PUSHSLAVEJOBUPDATES	Enables job changes made on a grid slave to be pushed to the grid head or master. Without this flag, jobs being reported to the grid head do not show any changes made on the remote Moab server (via mjobctl and so forth).
REPORT	N/A
ROOTPROXYJOBSUBMISSION	Enables Admin proxy job submission, which means administrators may submit jobs in behalf of other users.
SHARED	Resources of this resource manager may be scheduled by multiple independent sources and may not be assumed to be owned by any single source.
SLAVEPEER	Information from this resource manager may not be used to identify new jobs or nodes. Instead, this information may only be used to update jobs and nodes discovered and loaded from other non-slave resource managers.
STATIC	This resource manager only provides partial object information and this information does not change over time. Consequently, this resource manager may only be called once per object to modify job and node information.

Example

# resource manager 'torque' should use asynchronous job start 
# and report resources in 'grid' mode
RMCFG[torque] FLAGS=asyncstart,grid

13.2.3 Scheduler/Resource Manager Interactions

In the simplest configuration, Moab interacts with the resource manager using the following four primary functions:

GETJOBINFO

Collect detailed state and requirement information about idle, running, and recently completed jobs.

GETNODEINFO

Collect detailed state information about idle, busy, and defined nodes.

STARTJOB

Immediately start a specific job on a particular set of nodes.

CANCELJOB

Immediately cancel a specific job regardless of job state.

Using these four simple commands, Moab enables nearly its entire suite of scheduling functions. More detailed information about resource manager specific requirements and semantics for each of these commands can be found in the specific resource manager (LL, PBS, or WIKI) overviews.

In addition to these base commands, other commands are required to support advanced features such as dynamic job support, suspend/resume, gang scheduling, and scheduler initiated checkpoint restart.

Information on creating a new scheduler resource manager interface can be found in the Adding New Resource Manager Interfaces section.