| Moab Workload Manager® |
3.5 Credential Overview
Moab supports the concept of credentials, which provide a means of attributing policy and resource access to entities such as users and groups. These credentials allow specification of job ownership, tracking of resource usage, enforcement of policies, and many other features. There are five types of credentials—user, group, account, class, and QoS. While the credentials have many similarities, each plays a slightly different role.
- 3.5.1 General Credential Attributes
- 3.5.2 User Credential
- 3.5.3 Group Credential
- 3.5.4 Account (or Project) Credential
- 3.5.5 Class (or Queue) Credential
- 3.5.6 QoS Credential
3.5.1 General Credential Attributes
Internally, credentials are maintained as objects. Credentials can be created, destroyed, queried, and modified. They are associated with jobs and requests providing access and privileges. Each credential type has the following attributes:
- Priority Settings
- Usage Limits
- Service Targets
- Credential and Partition Access
- Statistics
- Credential Defaults, State and Configuration Information
All credentials represent a form of identity, and when applied to a job, express ownership. Consequently, jobs are subject to policies and limits associated with their owners.
3.5.1.1 Credential Priority Settings
Each credential may be assigned a priority using the PRIORITY attribute. This priority affects a job's total credential priority factor as described in the Priority Factors section. In addition, each credential may also specify priority weight offsets, which adjust priority weights that apply to associated jobs. These priority weight offsets include FSWEIGHT, QTWEIGHT, and XFWEIGHT.
Example
# set priority weights CREDWEIGHT 1 USERWEIGHT 1 CLASSWEIGHT 1 SERVICEWEIGHT 1 XFACTORWEIGHT 10 QUEUETIMEWEIGHT 1000 # set credential priorities USERCFG[john] PRIORITY=200 CLASSCFG[batch] PRIORITY=15 CLASSCFG[debug] PRIORITY=100 QOSCFG[bottomfeeder] QTWEIGHT=-50 XFWEIGHT=100 ACCOUNTCFG[topfeeder] PRIORITY=100
3.5.1.2 Credential Usage Limits
Usage limits constrain which jobs may run, which jobs may be considered for scheduling, and what quantity of resources each individual job may consume. With usage limits, policies such as MAXJOB, MAXNODE, and MAXMEM may be enforced against both idle and active jobs. Limits may be applied in any combination as shown in the example below where usage limits include 32 active processors per group and 12 active jobs for user john. For a job to run, it must satisfy the most limiting policies of all associated credentials. The Throttling Policy section documents credential usage limits in detail.
3.5.1.3 Service Targets
Credential service targets allow jobs to obtain special treatment to meet usage or response time based metrics. Additional information about service targets can be found in the Fairshare section.
3.5.1.4 Credential and Partition Access
Access to partitions and to other credentials may be specified on a per credential basis with credential access lists, default credentials, and credential membership lists.
You can use the ALIST, PLIST, and QLIST attributes (shown in the following table) to specify the list of credentials or partitions that a given credential may access.
| Credential | Attribute |
|---|---|
| Account | ALIST (allows credential to access specified list of accounts) |
| Partition | PLIST (allows credential to access specified list of partitions) |
| QoS | QLIST (allows credential to access specified list of QoS's) |
Example
USERCFG[bob] ALIST=jupiter,quantum USERCFG[steve] ALIST=quantum
 |
Account-based access lists are only enforced if using an allocation manager or if the ENFORCEACCOUNTACCESS parameter is set to TRUE. |
Use the the *DEF attribute (shown in the following table) to specify the default credential or partition for a particular credential.
| Credential | Attribute |
|---|---|
| Account | ADEF (specifies default account) |
| Class | CDEF (specifies default class) |
| QoS | QDEF (specifies default QoS) |
Example
# user bob can access accounts a2, a3, and a6. If no account is explicitly requested, # his job will be assigned to account a3 USERCFG[bob] ALIST=a2,a3,a6 ADEF=a3 # user steve can access accounts a14, a7, a2, a6, and a1. If no account is explicitly # requested, his job will be assigned to account a2 USERCFG[steve] ALIST=a14,a7,a2,a6,a1 ADEF=a2
Specifying Credential Membership Lists
As an alternate to specifying access lists, administrators may also specify membership lists. This allows a credential to specify who can access it rather than allowing each credential to specify which credentials it can access. Membership lists are controlled using the MEMBERULIST, EXCLUDEUSERLIST and REQUIREDUSERLIST attributes, shown in the following table:
| Credential | Attribute |
|---|---|
| User | --- |
| Account, Group, QoS | MEMBERULIST |
| Class | EXCLUDEUSERLIST and REQUIREDUSERLIST |
Example
# account omega3 can only be accessed by users johnh, stevek, jenp ACCOUNTCFG[omega3] MEMBERULIST=johnh,stevek,jenp
Example 1: Controlling Partition Access on a Per User Basis
A site may specify the user john may access partitions atlas, pluto, and zeus and will default to partition pluto. To do this, include the following line in the configuration file:
USERCFG[john] PLIST=atlas,pluto,zeus PDEF=pluto
Example 2: Controlling QoS Access on a Per Group Basis
A site may also choose to allow everyone in the group staff to access QoS standard and special with a default QoS of standard. To do this, include the following line in the configuration file:
GROUPCFG[staff] QLIST=standard,special QDEF=standard
Example 3: Controlling Resource Access on a Per Account Basis
An organization wants to allow everyone in the account omega3 to access nodes 20 through 24. To do this, include the following in the configuration file:
ACCOUNTCFG[omega3] MEMBERULIST=johnh,stevek,jenp SRCFG[omega3] HOSTLIST=r:20-24 ACCOUNTLIST=omega3
3.5.1.5 Credential Statistics
Full statistics are maintained for each credential instance. These statistics record current and historical resource usage, level of service delivered, accuracy of requests, and many other aspects of workload. Note, though, that you must explicitly enable credential statistics as they are not tracked by default. You can enable credential statistics by including the following in the configuration file:
USERCFG[DEFAULT] ENABLEPROFILING=TRUE GROUPCFG[DEFAULT] ENABLEPROFILING=TRUE ACCOUNTCFG[DEFAULT] ENABLEPROFILING=TRUE CLASSCFG[DEFAULT] ENABLEPROFILING=TRUE QOSCFG[DEFAULT] ENABLEPROFILING=TRUE
3.5.1.6 Job Defaults, Credential State, and General Configuration
Credentials may apply defaults and force job configuration settings via the following parameters:
| COMMENT | |||
| Description: | Associates a comment string with the target credential. | ||
| Example: | USERCFG[steve] COMMENT='works for boss, provides good service' CLASSCFG[i3] COMMENT='queue for I/O intensive workload' | ||
| HOLD | |||
| Description: | Specifies a hold should be placed on all jobs associated with the target credential.
| ||
| Example: | GROUPCFG[bert] HOLD=yes | ||
| JOBFLAGS | |||
| Description: | Assigns the specified job flag to all jobs with the associated credential. | ||
| Example: | CLASSCFG[batch] JOBFLAGS=suspendable QOSCFG[special] JOBFLAGS=restartable | ||
| NOSUBMIT | |||
| Description: | Specifies whether jobs belonging to this credential can submit jobs using msub. | ||
| Example: | ACCOUNTCFG[general] NOSUBMIT=TRUE CLASSCFG[special] NOSUBMIT=TRUE | ||
| OVERRUN | |||
| Description: | Specifies the amount of time a job may exceed its wallclock limit before being terminated. (Only applies to user and class credentials.) | ||
| Example: | CLASSCFG[bigmem] OVERRUN=00:15:00 | ||
| VARIABLE | |||
| Description: | Specifies attribute-value pairs associated with the specified credential. These variables may be used in triggers and other interfaces to modify system behavior. | ||
| Example: | GROUPCFG[staff] VARIABLE='nocharge=true' | ||
Credentials may carry additional configuration information. They may specify that detailed statistical profiling should occur, that submitted jobs should be held, or that corresponding jobs should be marked as preemptible.
3.5.2 User Credential
The user credential is the fundamental credential within a workload manager; each job requires an association with exactly one user. In fact, the user credential is the only required credential in Moab; all others are optional. In most cases, the job's user credential is configured within or managed by the operating system itself, although Moab may be configured to obtain this information from an independent security and identity management service.
As the fundamental credential, the user credential has a number of unique attributes.
3.5.2.1 Role
Moab supports role-based authorization, mapping particular roles to collections of specific users. See the Security section for more information.
3.5.2.1 Email Address
Facilities exist to allow user notification in the event of job or system failures or under other general conditions. This attribute allows these notifications to be mailed directly to the target user.
USERCFG[sally] EMAILADDRESS=sally@acme.com
3.5.2.2 Disable Moab User Email
You can disable Moab email notifications for a specific user.
USERCFG[john] NOEMAIL=TRUE
3.5.3 Group Credential
The group credential represents an aggregation of users. User-to-group mappings are often specified by the operating system or resource manager and typically map to a user's UNIX group ID. However, user-to-group mappings may also be provided by a security and identity management service, or you can specify such directly within Moab.
With many resource managers such as TORQUE, PBSPro, and LSF, the group associated with a job is either the user's active primary group as specified within the operating system or a group that is explicitly requested at job submission time. When a secondary group is requested, the user's default group and associated policies are not taken into account. Also note that a job may only run under one group. If more constraining policies are required for these systems, an alternate aggregation scheme such as the use of Account or QOS credentials is recommended.
To submit a job as a secondary group, refer to your local resource manager's job submission options. For TORQUE users, see the group_list=g_list option of the qsub -W command.
3.5.4 Account Credential
The account credential is also referred to as the project. This credential is generally associated with a group of users along the lines of a particular project for accounting and billing purposes. User-to-accounting mapping may be obtained from a resource manager or allocation manager, or you can configure it directly within Moab. Access to an account can be controlled via the ALIST and ADEF credential attributes specified via the Identity Manager or the moab.cfg file.
The MANAGERS attribute (applicable only to the account and class credentials) allows an administrator to assign a user the ability to manage jobs inside the crendential, as if the user is the job owner.
Example: MANAGERS Attribute
ACCOUNTCFG[general] MANAGERS=ops ACCOUNTCFG[special] MANAGERS=stevep
If a user is able to access more than one account, the desired account can be specified at job submission time using the resource-manager specific attribute. For example, with TORQUE this is accomplished using the -A argument to the qsub command.
Example: Enforcing Account Usage
Job-to-account mapping can be enforced using the ALIST attribute and the ENFORCEACCOUNTACCESS parameter.
USERCFG[john] ALIST=proj1,proj3 USERCFG[steve] ALIST=proj2,proj3,proj4 USERCFG[brad] ALIST=proj1 USERCFG[DEFAULT] ALIST=proj2 ENFORCEACCOUNTACCESS TRUE ...
3.5.5 Class Credential
- 3.5.5.1 Class Job Defaults
- 3.5.5.2 Per Job Min/Max Limits
- 3.5.5.3 Resource Access
- 3.5.5.4 Class Membership Constraints
- 3.5.5.5 Attributes Enabling Class Access to Other Credentials
- 3.5.5.6 Special Class Attributes (such as Managers and Job Prologs)
- 3.5.5.7 Setting Default Classes
- 3.5.5.8 Creating a Remap Class
- 3.5.5.9 Class Attribute Overview
- 3.5.5.10 Enabling Queue Complex Functionality
The concept of the class credential is derived from the resource manager class or queue object. Classes differ from other credentials in that they more directly impact job attributes. In standard HPC usage, a user submits a job to a class and this class imposes a number of factors on the job. The attributes of a class may be specified within the resource manager or directly within Moab. Class attributes include the following:
- Job Defaults
- Per Job Min/Max Limits
- Resource Access Constraints
- Class Membership Constraints
- Attributes Enabling Class Access to Other Credentials
- Special Class Attributes
|
When using SLURM, Moab classes have a one-to-one relationship with SLURM partitions of the same name. |
|
For all classes configured in Moab, a resource manager queue with the same name should be created. |
3.5.5.1 Class Job Defaults
Classes can be assigned to a default job template that can apply values to job attributes not explicitly specified by the submitter. Additionally, you can specify shortcut attributes from the table that follows:
| Attribute | Description |
|---|---|
| DEFAULT.ATTR | Job Attribute |
| DEFAULT.DISK | Required Disk (in MB) |
| DEFAULT.EXT | Job RM Extension |
| DEFAULT.FEATURES | Required Node Features/Properties |
| DEFAULT.GRES | Required Consumable Generic Resources |
| DEFAULT.MEM | Required Memory/RAM (in MB) |
| DEFAULT.NODE | Required Node Count |
| DEFAULT.NODESET | Node Set Specification |
| DEFAULT.PROC | Required Processor Count |
| DEFAULT.TPN | Tasks Per Node |
| DEFAULT.WCLIMIT | Wallclock Limit |
|
Defaults set in a class/queue of the resource manager will override the default values of the corresponding class/queue specified in Moab. |
|
RESOURCELIMITPOLICY must be configured in order for the CLASSCFG limits to take effect. |
Example
CLASSCFG[batch] DEFAULT.DISK=200MB DEFAULT.FEATURES=prod DEFAULT.WCLIMIT=1:00:00 CLASSCFG[debug] DEFAULT.FEATURES=debug DEFAULT.WCLIMIT=00:05:00
3.5.5.2 Per Job Min/Max Limits
Classes can be assigned a minimum and a maximum job template that constrains resource requests. Jobs submitted to a particular queue must meet the resource request constraints of these templates.
| Limit | Description |
|---|---|
| MAX.CPUTIME | Max Allowed Utilized CPU Time |
| MAX.NODE | Max Allowed Node Count |
| MAX.PROC | Max Allowed Processor Count |
| MAX.PS | Max Requested Processor-Seconds |
| MIN.NODE | Min Allowed Node Count |
| MIN.PROC | Min Allowed Processor Count |
| MIN.PS | Min Requested Processor-Seconds |
| MIN.TPN | Min Tasks Per Node |
| MIN.WCLIMIT | Min Requested Wallclock Limit |
| MAX.WCLIMIT | Max Requested Wallclock Limit |
|
The parameters listed in the preceding table are for classes only, and they function on a per-job basis. The MAX.* and MIN.* parameters are different from the MAXJOB, MAXNODE, and MAXMEM parameters described earlier in Credential Usage Limits. |
3.5.5.3 Resource Access
Classes may be associated with a particular set of compute resources. Consequently, jobs submitted to a given class may only use listed resources. This may be handled at the resource manager level or via the CLASSCFG HOSTLIST attribute.
3.5.5.4 Class Membership Constraints
Classes may be configured at either the resource manager or scheduler level to only allow select users and groups to access them. Jobs that do not meet these criteria are rejected. If specifying class membership/access at the resource manager level, see the respective resource manager documentation. Moab automatically detects and enforces these constraints. If specifying class membership/access at the scheduler level, use the REQUIREDUSERLIST or EXCLUDEUSERLIST attributes of the CLASSCFG parameter.
|
Under most resource managers, jobs must always be a member of one and only one class. |
3.5.5.5 Attributes Enabling Class Access to Other Credentials
Classes may be configured to allow jobs to access other credentials such as QoS's and Accounts. This is accomplished using the QDEF, QLIST, ADEF, and ALIST attributes.
3.5.5.6 Special Class Attributes
The class object also possesses a few unique attributes including JOBPROLOG, JOBEPILOG, JOBTRIGGER, RESFAILPOLICY, and DISABLEAM attributes described in what follows:
MANAGERS
Users listed via the MANAGERS parameter are granted full control over all jobs submitted to or running within the specified class.
# allow john and steve to cancel and modify all jobs submitted to the class/queue special CLASSCFG[special] MANAGERS=john,steve
In particular, a class manager can perform the following actions on jobs within a class/queue:
- view/diagnose job (checkjob)
- cancel, requeue, suspend, resume, and checkpoint job (mjobctl)
- modify job (mjobctl)
The JOBPROLOG class performs a function similar to the resource manager level job prolog feature; however, there are some key differences:
- Moab prologs execute on the head node; resource manager prologs execute on the nodes allocated to the job.
- Moab prologs execute as the primary Moab administrator, resource manager prologs execute as root.
- Moab prologs can incorporate cluster environment information into their decisions and actions. (See Valid Variables.)
- Unique Moab prologs can be specified on a per class basis.
- Job start requests are not sent to the resource manager until the Moab job prolog is successfully completed.
- Error messages generated by a Moab prolog are attached to jobs and associated objects; stderr from prolog script is attached to job.
- Moab prologs have access to Moab internal and peer services.
Valid epilog and prolog variables are:
- $TIME - Time that the trigger launches
- $HOME - Moab home directory
- $USER - User name the job is running under
- $JOBID - Unqiue job identifier
- $HOSTLIST - Entire host list for job
- $MASTERHOST - Master host for job
The JOBPROLOG class attribute allows a site to specify a unique per-class action to take before a job is allowed to start. This can be used for environmental provisioning, pre-execution resource checking, security management, and other functions. Sample uses may include enabling a VLAN, mounting a global file system, installing a new application or virtual node image, creating dynamic storage partitions, or activating job specific software services.
|
A prolog is considered to have failed if it returns a negative number. If a prolog fails, the associated job will not start. |
|
If a prolog executes successfully, the associated epilog is guaranteed to start, even if the job fails for any reason. This allows the epilog to undo any changes made to the system by the prolog. |
Job Prolog Examples
# explicitly specify prolog arguments for special epilog CLASSCFG[special] JOBPROLOG='$TOOLSDIR/specialprolog.pl $JOBID $HOSTLIST' # use default prolog arguments for batch prolog CLASSCFG[batch] JOBPROLOG=$TOOLSDIR/batchprolog.pl
JOBEPILOG
The Moab epilog is nearly identical to the prolog in functionality except that it runs after the job completes within the resource manager but before the scheduler releases the allocated resources for use by subsequent jobs. It is commonly used for job clean-up, file transfers, signalling peer services, and undoing other forms of resource customization.
|
An epilog is considered to have failed if it returns a negative number. If an epilog fails, the associated job will be annotated and a message will be sent to administrators. |
Job triggers can be directly associated with jobs submitted into a class using the JOBTRIGGER attribute. Job triggers are described using the standard trigger description language specified in the Trigger overview section. In the example that follows, users submitting jobs to the class debug will be notified with a descriptive message anytime their job is preempted.
CLASSCFG[batch] JOBTRIGGER=atype=exec,etype=preempt,action="$HOME/tools/preemptnotify.pl $OID $OWNER $HOSTNAME"
This policy allows specification of the action to take on a per-class basis when a failure occurs on a node allocated to an actively running job. See the Node Availability Overview for more information.
You can disable allocation management for jobs in specific classes by setting the DISABLEAM class attribute to FALSE. For all jobs outside of the specified classes, allocation enforcement will continue to be enforced.
# do not enforce allocations on low priority and debug jobs CLASSCFG[lowprio] DISABLEAM=TRUE CLASSCFG[debug] DISABLEAM=TRUE
3.5.5.7 Setting Default Classes
In many cases, end-users do not want to be concerned with specifying a job class/queue. This is often handled by defining a default class. Whenever a user does not explicitly submit a job to a particular class, a default class, if specified, is used. In resource managers such as TORQUE, this can be done at the resource manager level and its impact is transparent to the scheduler. The default class can also be enabled within the scheduler on a per resource manager or per user basis. To set a resource manager default class within Moab, use the DEFAULTCLASS attribute of the RMCFG parameter. For per user defaults, use the CDEF attribute of the USERCFG parameter.
3.5.5.8 Creating a Remap Class
If a single default class is not adequate, Moab provides more flexible options with the REMAPCLASS parameter. If this parameter is set and a job is submitted to the remap class, Moab attempts to determine the final class to which a job belongs based on the resources requested. If a remap class is specified, Moab compares the job's requested nodes, processors, memory, and node features with the class's corresponding minimum and maximum resource limits. Classes are searched in the order in which they are defined; when the first match is found, Moab assigns the job to that class. In the example that follows, a job requesting 4 processors and the node feature fast are assigned to the class quick.
# jobs submitted to 'batch' should be remapped REMAPCLASS batch # stevens only queue CLASSCFG[stevens] REQ.FEATURES=stevens REQUIREDUSERLIST=stevens,stevens2 # special queue for I/O nodes CLASSCFG[io] MAX.PROC=8 REQ.FEATURES=io # general access queues CLASSCFG[quick] MIN.PROC=2 MAX.PROC=8 REQ.FEATURES=fast|short CLASSCFG[medium] MIN.PROC=2 MAX.PROC=8 CLASSCFG[DEFAULT] MAX.PROC=64 ...
The following parameters can be used to remap jobs to different classes:
- MIN.PROC
- MAX.PROC
- MIN.WCLIMIT
- MAX.WCLIMIT
- REQ.FEATURES
- REQ.FLAGS=INTERACTIVE
- REQUIREDUSERLIST
If the parameter REMAPCLASSLIST is set, then only the listed classes are searched and they are searched in the order specified by this parameter. If none of the listed classes are valid for a particular job, that job retains its original class.
|
The remap class only works with resource managers that allow dynamic modification of a job's assigned class/queue. Also note that OpenPBS and TORQUE 1.x support dynamic job queue modification, but this change is not persistent and will be lost if pbs_server is restarted. |
|
If default credentials are specified on a remap class, a job submitted to that class will inherit those credentials. If the destination class has different defaults credentials, the new defaults override the original settings. If the destination class does not have default credentials, the job maintains the defaults inherited from the remap class. |
3.5.5.9 Class Attribute Overview
The following table enumerates the different parameters for CLASSCFG:
| DEFAULT.ATTR | |||
| Format: | <ATTRIBUTE>[,<ATTRIBUTE>]... | ||
| Description: | One or more comma-delimited generic job attributes. | ||
| Example: | --- | ||
| DEFAULT.DISK | |||
| Format: | <INTEGER> | ||
| Description: | Default amount of requested disk space. | ||
| Example: | --- | ||
| DEFAULT.EXT | |||
| Format: | <STRING> | ||
| Description: | Default job RM extension. | ||
| Example: | --- | ||
| DEFAULT.FEATURES | |||
| Format: | Comma-delimited list of features. | ||
| Description: | Default list of requested node features (a.k.a, node properties). This only applies to compute resource reqs. | ||
| Example: | --- | ||
| DEFAULT.GRES | |||
| Format: | <STRING>[<COUNT>][,<STRING>[<COUNT>]]... | ||
| Description: | Default list of per task required consumable generic resources. | ||
| Example: | CLASSCFG[viz] DEFAULT.GRES=viz:2 | ||
| DEFAULT.MEM | |||
| Format: | <INTEGER> (in MB) | ||
| Description: | Default amount of requested memory. | ||
| Example: | --- | ||
| DEFAULT.NODE | |||
| Format: | <INTEGER> | ||
| Description: | Default required node count. | ||
| Example: | --- | ||
| DEFAULT.NODESET | |||
| Format: | <SETTYPE>:<SETATTR>[:<SETLIST>[,<SETLIST>]...] | ||
| Description: | Default node set. | ||
| Example: | CLASSCFG[amd] DEFAULT.NODESET=ONEOF:FEATURE:ATHLON,OPTERON | ||
| DEFAULT.PROC | |||
| Format: | <INTEGER> | ||
| Description: | Default number of requested processors. | ||
| Example: | --- | ||
| DEFAULT.TPN | |||
| Format: | <INTEGER> | ||
| Description: | Default number of tasks per node. | ||
| Example: | --- | ||
| DEFAULT.WCLIMIT | |||
| Format: | <INTEGER> | ||
| Description: | Default wallclock limit. | ||
| Example: | --- | ||
| EXCL.FEATURES | |||
| Format: | Comma- or pipe-delimited list of node features. | ||
| Description: | Set of excluded (disallowed) features. If delimited by commas, reject job if all features are requested; if delimited by the pipe symbol (|), reject job if at least one feature is requested. | ||
| Example: | CLASSCFG[intel] EXCL.FEATURES=ATHLON,AMD | ||
| EXCL.FLAGS | |||
| Format: | Comma-delimited list of job flags. | ||
| Description: | Set of excluded (disallowed) job flags. Reject job if any listed flags are set. | ||
| Example: | CLASSCFG[batch] EXCL.FLAGS=INTERACTIVE | ||
| EXCLUDEUSERLIST | |||
| Format: | Comma-delimited list of users. | ||
| Description: | List of users not permitted access to class.
| ||
| Example: | --- | ||
| FORCENODEACCESSPOLICY | |||
| Format: | one of SINGLETASK, SINGLEJOB, SINGLEUSER, or SHARED | ||
| Description: | Node access policy associated with queue. If set, this value overrides any per job settings specified by the user at the job level. (See Node Access Policy overview for more information.) | ||
| Example: | CLASSCFG[batch] FORCENODEACCESSPOLICY=SINGLEJOB | ||
| FSCAP | |||
| Format: | <DOUBLE>[%] | ||
| Description: | See fairshare policies specification. | ||
| Example: | --- | ||
| FSTARGET | |||
| Format: | <DOUBLE>[%] | ||
| Description: | See fairshare policies specification. | ||
| Example: | --- | ||
| HOSTLIST | |||
| Format: | Host expression, or comma-delimited list of hosts or host ranges. | ||
| Description: | List of hosts associated with a class. If specified, Moab constrains the availability of a class to only nodes listed in the class host list. | ||
| Example: | CLASSCFG[batch] HOSTLIST=r:abs[45-113] | ||
| JOBEPILOG | |||
| Format: | <STRING> | ||
| Description: | Scheduler level job epilog to be run after job is completed by resource manager. (See special class attributes.) | ||
| Example: | --- | ||
| JOBFLAGS | |||
| Format: | Comma-delimited list of job flags. | ||
| Description: | See the flag overview for a description of legal flag values. | ||
| Example: | CLASSCFG[batch] JOBFLAGS=restartable | ||
| JOBPROLOG | |||
| Format: | <STRING> | ||
| Description: | Scheduler level job prolog to be run before job is started by resource manager. (See special class attributes.) | ||
| Example: | --- | ||
| JOBTRIGGER | |||
| Format: | <STRING> | ||
| Description: | Scheduler level job trigger to be associated with jobs submitted to this class. (See special class attributes.) | ||
| Example: | --- | ||
| MANAGERS | |||
| Format: | <USER>[,<USER>]... | ||
| Description: | Users allowed to control, cancel, preempt, and modify jobs within class/queue. (See special class attributes.) | ||
| Example: | CLASSCFG[fast] MANAGERS=root,kerry,e43 | ||
| MAXJOB | |||
| Format: | <INTEGER> | ||
| Description: | Maximum number of jobs allowed in the class. | ||
| Example: | --- | ||
| MAXPROCPERNODE | |||
| Format: | <INTEGER> | ||
| Description: | Maximum number of processors requested per node. | ||
| Example: | --- | ||
| MAX.CPUTIME | |||
| Format: | <INTEGER> | ||
| Description: | Maximum allowed utilized CPU time. | ||
| Example: | --- | ||
| MAX.NODE | |||
| Format: | <INTEGER> | ||
| Description: | Maximum number of requested nodes per job. (Also used when REMAPCLASS is set to correctly route the job.) | ||
| Example: | CLASSCFG[batch] MAX.NODE=64 | ||
| MAX.PROC | |||
| Format: | <INTEGER> | ||
| Description: | Maximum number of requested processors per job. (Also used when REMAPCLASS is set to correctly route the job.) | ||
| Example: | CLASSCFG[small] MAX.PROC[USER]=3,6 | ||
| MAX.PS | |||
| Format: | <INTEGER> | ||
| Description: | Maximum requested processor-seconds. | ||
| Example: | --- | ||
| MAX.WCLIMIT | |||
| Format: | [[[DD:]HH:]MM:]SS | ||
| Description: | Maximum allowed wallclock limit per job. (Also used when REMAPCLASS is set to correctly route the job.) | ||
| Example: | CLASSCFG[long] MAX.WCLIMIT=96:00:00 | ||
| MIN.NODE | |||
| Format: | <INTEGER> | ||
| Description: | Minimum number of requested nodes per job. (Also used when REMAPCLASS is set to correctly route the job.) | ||
| Example: | CLASSCFG[dev] MIN.NODE=16 | ||
| MIN.PROC | |||
| Format: | <INTEGER> | ||
| Description: | Minimum number of requested processors per job. (Also used when REMAPCLASS is set to correctly route the job.) | ||
| Example: | CLASSCFG[dev] MIN.PROC=32 | ||
| MIN.PS | |||
| Format: | <INTEGER> | ||
| Description: | Minimum requested processor-seconds. | ||
| Example: | --- | ||
| MIN.TPN | |||
| Format: | <INTEGER> | ||
| Description: | Minimum required tasks per node per job. (Also used when REMAPCLASS is set to correctly route the job.) | ||
| Example: | --- | ||
| MIN.WCLIMIT | |||
| Format: | [[[DD:]HH:]MM:]SS | ||
| Description: | Minimum required wallclock limit per job. (Also used when REMAPCLASS is set to correctly route the job.) | ||
| Example: | --- | ||
| NODEACCESSPOLICY | |||
| Format: | one of SINGLETASK, SINGLEJOB, SINGLEUSER, or SHARED | ||
| Description: | Default node access policy associated with queue. This value will be overridden by any per job settings specified by the user at the job level. (See Node Access Policy overview.) | ||
| Example: | CLASSCFG[batch] NODEACCESSPOLICY=SINGLEJOB | ||
| PARTITION | |||
| Format: | <STRING> | ||
| Description: | Partition name where jobs associated with this class must run. | ||
| Example: | CLASSCFG[batch] PARTITION=p12 | ||
| PRIORITY | |||
| Format: | <INTEGER> | ||
| Description: | Priority associated with the class. (See Priority overview.) | ||
| Example: | CLASSCFG[batch] PRIORITY=1000 | ||
| QDEF | |||
| Format: | <QOSID> | ||
| Description: | Default QoS for jobs submitted to this class. | ||
| Example: | CLASSCFG[batch] QDEF=base | ||
| QLIST | |||
| Format: | <QOSID>[,<QOSID>]... | ||
| Description: | List of accessible QoS's for jobs submitted to this class. | ||
| Example: | CLASSCFG[batch] QDEF=base QLIST=base,fast,special,bigio | ||
| REQ.FEATURES | |||
| Format: | Comma- or pipe-delimited list of node features. | ||
| Description: | Set of required features. If delimited by commas, all features are required; if delimited by the pipe symbol (|), at least one feature is required. | ||
| Example: | CLASSCFG[amd] REQ.FEATURES=ATHLON,AMD | ||
| REQ.FLAGS | |||
| Format: | REQ.FLAGS can be used with only the INTERACTIVE flag. | ||
| Description: | Sets the INTERACTIVE flag on jobs in this class. | ||
| Example: | CLASSCFG[orion] REQ.FLAGS=INTERACTIVE | ||
| REQUIREDACCOUNTLIST | |||
| Format: | Comma-delimited list of accounts. | ||
| Description: | List of accounts allowed to access and use a class (analogous to *LIST for other credentials).
| ||
| Example: | CLASSCFG[jasper] REQUIREDACCOUNTLIST=testers,development | ||
| REQUIREDUSERLIST | |||
| Format: | Comma-delimited list of users. | ||
| Description: | List of users allowed to access and use a class (analogous to *LIST for other credentials).
| ||
| Example: | CLASSCFG[jasper] REQUIREDUSERLIST=john,u13,steve,guest | ||
| REQUIREDQOSLIST | |||
| Format: | Comma-delimited list of QoS's | ||
| Description: | List of QoS's allowed to access and use a class (analogous to *LIST for other credentials).
|
||
| Example: |
CLASSCFG[jasper] REQUIREDQOSLIST=hi,lo |
||
| RMLIST | |||
| Format: | [!]<RMID>[,[!]<RMID>]]... | ||
| Description: | List of resource managers that can (or cannot) view or access the class. By default, all resource managers can view and access all queues/classes. If this attribute is specified, only listed resource managers can see the associated queue. If an exclamation point character (!) is specified in the value, then access is granted to all resource managers who are not listed. This feature is most commonly used in grid environments. | ||
| Example: | CLASSCFG[special] RMLIST=LL,chemgrid | ||
| SYSPRIO | |||
| Format: | <INTEGER> | ||
| Description: | Value of system priority applied to every job submitted to this class. | ||
| Example: | CLASSCFG[special] SYSPRIO=100 | ||
| WCOVERRUN | |||
| Format: | [[[DD:]HH:]MM:]SS | ||
| Description: | Tolerated amount of time beyond the specified wallclock limit. | ||
| Example: | --- | ||
3.5.5.10 Enabling Queue Complex Functionality
Queue complexes allow an organization to build a hierarchy of queues and apply certain limits and rules to collections of these queues. Moab supports this functionality in two ways. The first way, queue mapping, is very simple but limited in functionality. The second method provides very rich functionality but requires more extensive configuration using the Moab hierarchical fairshare facility.
Queue Mapping
Queue mapping allows collections of queues to be mapped to a parent credential object against which various limits and policies can be applied, as in the following example.
QOSCFG[general] MAXIJOB[USER]=14 PRIORITY=20 QOSCFG[prio] MAXIJOB[USER]=8 PRIORITY=2000 # group short, med, and long jobs into 'general' QOS CLASSCFG[short] QDEF=general FSTARGET=30 CLASSCFG[med] QDEF=general FSTARGET=40 CLASSCFG[long] QDEF=general FSTARGET=30 MAXPROC=200 # group interactive and debug jobs into 'prio' QOS CLASSCFG[inter] QDEF=prio CLASSCFG[debug] QDEF=prio CLASSCFG[premier] PRIORITY=10000
3.5.6 QoS Credential
The concept of a quality of service (QoS) credential is unique to Moab and is not derived from any underlying concept or peer service. In most cases, the QoS credential is used to allow a site to set up a selection of service levels for end-users to choose from on a long-term or job-by-job basis. QoS's differ from other credentials in that they are centered around special access where this access may allow use of additional services, additional resources, or improved responsiveness. Unique to this credential, organizations may also choose to apply different charge rates to the varying levels of service available within each QoS. As QoS is an internal credential, all QoS configuration occurs within Moab.
QoS access and QoS defaults can be mapped to users, groups, accounts, and classes, allowing limited service offering for key users. As mentioned, these services focus around increasing access to special scheduling capabilities & additional resources and improving job responsiveness. At a high level, unique QoS attributes can be broken down into the following:
3.5.6.1 QoS Usage Limit Overrides
All credentials allow specification of job limits. In such cases, jobs are constrained by the most limiting of all applicable policies. With QoS override limits, however, jobs are limited by the override, regardless of other limits specified.
3.5.6.2 QoS Service Targets
Service targets cause the scheduler to take certain job-related actions as various responsiveness targets are met. Targets can be set for either job queue time or job expansion factor and cause priority adjustments, reservation enforcement, or preemption activation. In strict service centric organizations, Moab can be configured to trigger various events and notifications in the case of failure by the cluster to meet responsiveness targets.
3.5.6.3 QoS Privilege Flags
QoS's can provide access to special capabilities. These capabilities include preemption, job deadline support, backfill, next to run priority, guaranteed resource reservation, resource provisioning, dedicated resource access, and many others. See the complete list in the QoS Facility Overview section.
3.5.6.4 QoS Charge Rate
Associated with the QoS's many privileges is the ability to assign end-users costs for the use of these services. This charging can be done on a per-QoS basis and may be specified for both dedicated and use-based resource consumption. The Per QoS Charging section covers more details on QoS level costing configuration while the Charging and Allocation Management section provides more details regarding general single cluster and multi-cluster charging capabilities.
3.5.6.5 QoS Access Controls
QoS access control can be enabled on a per QoS basis using the MEMBERULIST attribute or specified on a per-requestor basis using the QDEF and QLIST attributes of the USERCFG, GROUPCFG, ACCOUNTCFG, and CLASSCFG parameters. See Managing QoS Access for more detail.
See Also