| Moab Workload Manager® |
7.3 Quality of Service (QoS) Facilities
This section describes how to do the following:
- Allow key projects access to special services (such as preemption, resource dedication, and advance reservations).
- Provide access to special resources by requested QoS.
- Enable special treatment within priority and fairshare facilities by requested QoS.
- Provide exemptions to usage limits and other policies by requested QoS.
- Specify delivered service and response time targets.
- Enable job deadline guarantees.
- Control the list of QoS's available to each user and job.
- Enable special charging rates based on requested or delivered QoS levels.
- Enable limits on the extent of use for each defined QoS.
- Monitor current and historical usage for each defined QoS.
- 7.3.1 QoS Overview
- 7.3.2 QoS Enabled Privileges
- 7.3.3 Managing QoS Access
- 7.3.4 Configuring QoS Specific Charging
- 7.3.5 Requesting QoS Services at Job Submission
7.3.1 QoS Overview
Moab's QoS facility allows a site to give special treatment to various classes of jobs, users, groups, and so forth. Each QoS object can be thought of as a container of special privileges ranging from fairness policy exemptions, to special job prioritization, to special functionality access. Each QoS object also has an extensive access list of users, groups, and accounts that can access these privileges.
|
| Moab Workload Manager - QoS Management |
Sites can configure various QoS's each with its own set of priorities, policy exemptions, and special resource access settings. They can then configure user, group, account, and class access to these QoS's. A given job will have a default QoS and may have access to several additional QoS's. When the job is submitted, the submitter may request a specific QoS or just allow the default QoS to be used. Once a job is submitted, a user may adjust the QoS of the job at any time using the setqos command. The setqos command will only allow the user to modify the QoS of that user's jobs and only change the QoS to a QoS that this user has access to. Moab administrators may change the QOS of any job to any value.
Jobs can be granted access to QoS privileges if the QoS is listed in the system default configuration QDEF (QoS default) or QLIST (QoS access list), or if the QoS is specified in the QDEF or QLIST of a user, group, account, or class associated with that job. Alternatively, a user may access QoS privileges if that user is listed in the QoS's MEMBERULIST attribute.
The mdiag -q command can be used to obtain information about the current QoS configuration including specified credential access.
7.3.2 QoS Enabled Privileges
The privileges enabled via QoS settings may be broken into the following categories:
- Special Prioritization
- Service Access and Constraints
- Override Policies and Policy Exemptions
All privileges are managed via the QOSCFG parameter.
7.3.2.1 Special Prioritization
| Attribute Name | Description |
| FSTARGET | Specifies QoS fairshare target. |
| FSWEIGHT | Sets QoS fairshare weight offset affecting a job's fairshare priority component. |
| PRIORITY | Assigns priority to all jobs requesting particular QoS. |
| QTTARGET | Sets QoS queuetime target affecting a job's target priority component and QoS delivered. |
| QTWEIGHT | Sets QoS queuetime weight offset affecting a job's service priority component. |
| XFTARGET | Sets QoS XFactor target affecting a job's target priority component and QoS delivered. |
| XFWEIGHT | Sets QoS XFactor weight offset affecting a job's service priority component. |
Example
7.3.2.2 Service Access and Constraints
The QoS facility can be used to enable special service and to disable default services. Most services are enabled/disabled by setting the QoS QFLAGS attribute.
| Flag Name | Description |
| COALLOC | Jobs may allocate resources from multiple partitions (span partitions) if and only if they are associated with a QoS that has this flag set and they explicitly set the COALLOC job flag. |
| DEADLINE | Job may request an absolute or relative completion deadline and Moab will reserve resources to meet that deadline. (An alternative priority based deadline behavior is discussed in the PRIORITY FACTORS section.) |
| DEDICATED | Moab dedicates all resources of an allocated node to the job meaning that the job will not share a node's compute resources with any other job. |
| DYNAMIC | Moab allows jobs to dynamically allocate/de-allocate resources while the job is actively running. (See Dynamic Jobs.) |
| ENABLEUSERRSV | Allow user or personal reservations to be created and managed. |
| IGNALL | Scheduler ignores all resource usage policies for jobs associated with this QoS. |
| NOBF | Job is not considered for backfill. |
| NORESERVATION | Job should never reserve resources regardless of priority. |
| NTR | Job is prioritized to run next. |
| PREEMPTCONFIG | User jobs may specify options to alter how preemption impacts the job such as minpreempttime. |
| PREEMPTEE | Job may be preempted by higher priority PREEMPTOR jobs. |
| PREEMPTFSV | Job may be preempted by higher priority PREEMPTOR jobs if it exceeds its fairshare target when started. |
| PREEMPTOR | Job may preempt lower priority PREEMPTEE jobs. |
| PREEMPTSPV | Job may be preempted by higher priority PREEMPTOR jobs if it currently violates a soft usage policy limit. |
| PROVISION | If the job cannot locate available resources with the needed OS or software, the scheduler may provision a number of nodes to meet the needed OS or software requirements. |
| RESERVEALWAYS | Job should create resource reservation regardless of job priority. |
| RSVCREATIONCOST | The base cost to users when creating reservations. |
| TRIGGER | The job is able to directly specify triggers. |
| USERESERVED[:<RSVID>] | Job may only use resources within accessible reservations. If <RSVID> is specified, job may only use resources within the specified reservation. |
Example 1: For Low QoS Job, Disable Backfill and Make Job Preemptible
Example 2: Bind All Jobs to Chemistry Reservation
Per QoS Required Reservations
If desired, jobs associated with a particular QoS can be locked into a reservation or reservation group using the REQRID attribute. For example, to force jobs using QoS jasper to only use the resources within the failsafe standing reservation, use the following:
Setting Per QoS Network Service Levels
Minimum network service levels can be specified using the MINBANDWIDTH and MAXLATENCY attributes. If specified, Moab does not allow a job to start until these criteria can be satisfied and allocates or reserves specific resources or combinations of resources to fulfill these constraints. Further, if the Network Management System supports it, Moab dynamically creates network fabric partitions with guaranteed service levels.
7.3.2.3 Usage Limits and Overrides
All credentials, including QoS, allow specification of job usage limits as described in the Basic Fairness Policies overview. In such cases, jobs are constrained by the most limiting of all applicable policies. With QoS's, an override limit may also be specified and with this limit, jobs are constrained by the override, regardless of other limits specified. In particular, the following policies may be overridden:
- MAXJOB
- MAXNODE
- MAXPE
- MAXPROC
- MAXPS
Usage limits are overridden by specifying the QoS attribute O<LIMIT>, as in OMAXJOB or OMAXPROC. To override system/job limits, use the OMAXJ<LIMIT>, as in OMAXJPS or OMAXJWC.
In addition to job usage limits, QoS's (as with other credentials), can also limit personal reservations. Attributes such as RMAXDURATION, RMAXPROC, and RMAXTOTALPROC can be used to constrain resources available to personal reservations. See Personal Reservation Usage Limits for more information.
Example
7.3.2.4 Service Access Thresholds
Jobs can be granted access to services such as preemption and reservation creation, and they can be granted access to resource reservations. However, with QoS thresholds, this access can be made conditional on the current queuetime and XFactor metrics of an idle job. The following table lists the available QoS service thresholds:
| Threshold Attribute | Description |
| PREEMPTQTTHRESHOLD | A job with this QoS becomes a preemptor if the specified queuetime threshold is reached. |
| PREEMPTXFTHRESHOLD | A job with this QoS becomes a preemptor if the specified XFactor threshold is reached. |
| RSVQTTHRESHOLD | A job with this QoS can create a job reservation to guarantee resource access if the specified queuetime threshold is reached. |
| RSVXFTHRESHOLD | A job with this QoS can create a job reservation to guarantee resource access if the specified XFactor threshold is reached. |
| ACLQTTHRESHOLD | A job with this QoS can access reservations with a corresponding QoS ACL only if the specified queuetime threshold is reached. |
| ACLXFTHRESHOLD | A job with this QoS can access reservations with a corresponding QoS ACL only if the specified XFactor threshold is reached. |
| TRIGGERQTTHRESHOLD | Uf a job with this QoS fails to run before this threshold is reached, any failure triggers associated with this QoS will fire. |
7.3.2.5 Preemption Management
Job preemption facilities can be controlled on a per-QoS basis using the PREEMPTEE and PREEMPTOR flags. Jobs that are preemptible can optionally be constrained to only be preempted in a particular manner by specifying the QoS PREEMPTPOLICY attribute as in the following example:
For preemption to be effective, a job must be marked as a preemptee and must be enabled for the requested preemption type. For example, if the PREEMPTIONPOLICY is set to suspend, a potential target job must be both a preemptee and marked with the job flag SUSPENDABLE. (See suspension for more information.) If the target job is not suspendable, it will be either requeued or canceled. Likewise, if the PREEMPTIONPOLICY is set to requeue, the job will be requeued if it is marked restartable. Otherwise, it will be canceled.
The minimum time a job must run before being considered eligible for preemption can also be configured on a per-QoS basis using the PREEMPTMINTIME parameter, which is analogous to the JOBPREEMPTMINACTIVETIME.
7.3.2.6 Privileged Resource Access
The RSVACCESSLIST and RSVPREEMPTLIST attributes specify a list of reservations or reservation groups that the requestor may access directly or is entitled to preempt in order to allocate needed resources.
This access is used in conjunction with the mshow and mrsvctl commands.
7.3.3 Managing QoS Access
Specifying Credential Based QoS Access
You can define the privileges allowed within a QoS by using the QOSCFG parameter; however, in most cases access to the QoS is enabled via credential specific *CFG parameters, specifically the USERCFG, GROUPCFG, ACCOUNTCFG, and CLASSCFG parameters, which allow defining QoS access lists and QoS defaults. Specify credential specific QoS access by using the QLIST and/or QDEF attributes of the associated credential parameter.
QOS Access via Logical OR
To enable QoS access, the QLIST and/or QDEF attributes of the appropriate user, group, account, or class/queue should be specified as in the following example:
By default, jobs may request a QoS if access to that QoS is allowed by any of the job's credentials. (In the previous example, a job from user john submitted to the class batch could request QoS's geo, chem, staff, or normal).
QOS Access via Logical AND
If desired, QoS access can be masked or logically ANDed if the QoS access list is specified with a terminating ampersand (&) as in the following example:
QoS access may also be specified from within the QoS object using the QoS MEMBERULIST attribute as in the following example:
NOTE: By default, if a job requests a QoS that it cannot access, Moab places a hold on that job. The QOSREJECTPOLICY can be used to modify this behavior.
7.3.4 Configuring QoS-Specific Charging
Charging policies and rates may be configured on a per-QoS basis. This capability provides a powerful way of allowing end-users to select the level of service they desire and the price they are willing to pay for that service.
Several policies allow control over exactly how a job gets charged; these include CHARGERATEPOLICY and CHARGEMETRICPOLICY. The first parameter, CHARGERATEPOLICY, allows a site to specify how jobs are charged for the resources they use.
A job's total cost can be adjusted on a per QoS basis by using one or more of the following attributes:
| QOS Attributes | Description |
| UTLRESCOST | Cost multiplier for used resources associated with using the specified QoS. |
| DEDRESCOST | Cost multiplier for dedicated resources associated with using the specified QoS. |
Example
NOTE: These charging parameters only impact internal charging job costs written to accounting records. They do not currently affect charges sent to an external allocation manager.
7.3.5 Requesting QoS Services at Job Submission
By default, jobs inherit a default QoS based on the user, group, class, and account associated with the job. If a job has access to multiple QoS levels, the submittor can explicitly request a particular QoS using the QoS resource manager extension as in the following example:
See Also