8.4 Preemption Policies
Many sites possess workloads of varying importance. While it may be critical that some jobs obtain resources immediately, other jobs are less turnaround time sensitive but have an insatiable hunger for compute cycles, consuming every available cycle for years on end. These latter jobs often have turnaround times on the order of weeks or months. The concept of cycle stealing, popularized by systems such as Condor, handles such situations well and enables systems to run low priority, preemptible jobs whenever something more pressing is not running. These other systems are often employed on compute farms of desktops where the jobs must vacate anytime interactive system use is detected.
8.4.1 Preemption Triggers
Preemption can be enabled in one of three ways. These include manual intervention, QOS based configuration, and use of the preemption based backfill algorithm.8.4.1.1 Admin Preemption Commands
The mjobctl command can be used to preempt jobs. Specifically, the command can be used to modify a job's execution state in the following ways:| Action |
Flag |
Details |
| Cancel |
-c |
terminate and remove job from queue |
| Checkpoint |
-C |
terminate and checkpoint job leaving job in queue |
| Requeue |
-R |
terminate job leaving job in queue |
| Resume |
-r |
resume suspended job |
| Start (execute) |
-x |
start idle job |
| Suspend |
-s |
suspend active job |
In general, users are allowed to suspend or terminate
jobs they own. Administrators are allowed to suspend, terminate, resume,
and execute any queued jobs.
8.4.1.2 QOS Based Preemption
Maui's QoS-based preemption system allows a site the ability to specify preemption rules and control access to preemption privileges. These abilities can be used to increase system throughput, improve job response time for specific classes of jobs, or other enable various political policies. All policies are enabled by specifying some QOS's with the flag PREEMPTOR , and other with the flag PREEMPTEE. For example, to enable a cycle stealing high throughput cluster, a QOS can be created for high priority jobs and marked with the flag PREEMPTOR; another QOS can be created for low priority jobs and marked with the flag PREEMPTEE . Finally, the RESERVATIONPOLICY parameter can be set to NEVER. With this configuration, low priority, preemptee jobs can be started whenever idle resources are available. These jobs will be allowed to run until a high priority job arrives, at which point the necessary low priority jobs will be preempted and the needed resources freed. This allows near immediate resource access for the high priority jobs. Using this approach, a cluster can maintain near 100% system utilization while still delivering excellent turnaround time to the jobs of greatest value.It is important to note the rules of QoS based preemption. Preemption only occurs when the following 3 conditions are satisfied:
- The preemptor job has the PREEMPTOR attribute set
- The preemptee job has the PREEMPTEE attribute set
- The preemptor job has a higher priority than the preemptee job
Use of the preemption system need not be limited to controlling low priority jobs. Other uses include optimistic scheduling and development job support.
Example:
----
PREEMPTPOLICY REQUEUE
QOSCFG[high] QFLAGS=PREEMPTOR
QOSCFG[med]
QOSCFG[low] QFLAGS=PREEMPTEE
----
The Moab Cluster ManagerTM's graphical interface presents numerous choices for configuration. For example, PREEMPTOR and PREEMPTEE attributes can be set when a QoS is created.
8.4.1.3 Preemption Based Backfill
The PREEMPT backfill policy allows a site to take advantage of optimistic scheduling. By default, backfill only allows jobs to run if they are guaranteed to have adequate time to run to completion. However, statistically, most jobs do not utilize their full requested wallclock limit. The PREEMPT backfill policy allows the scheduler to start backfill jobs even if required walltime is not available. If the job runs too long and interferes with another job which was guaranteed a particular timeslot, the backfill job is preempted and the priority job is allowed to run. When another potential timeslot becomes available, the preempted backfill job will again be optimistically executed. In environments with checkpointing or with poor wallclock accuracies, this algorithm has potential for significant savings. See the backfill section for more information.8.4.2 Types of Preemption
How the scheduler preempts a job is controlled by the PREEMPTPOLICY parameter. This parameter allows preemption to be enforced in one of the following manners:8.4.2.1 Job Requeue
Under this policy, active jobs are terminated and returned to the job queue in an idle state.8.4.2.2 Job Suspend
Suspend causes active jobs to stop executing but to remain in memory or the allocated compute nodes. While a suspended job frees up processor resources, it may continue to consume swap and/or other resources. Suspended jobs must be 'resumed' to continue executing. NOTE:If 'suspend' based preemption is selected, then the signal used to initiate the job suspend may be specified by setting the RM specific 'SUSPENDSIG' attribute, i.e. 'RMCFG[base] SUSPENDSIG=23'.8.4.2.3 Job Checkpoint
Systems which support job checkpointing allow a job to save off its current state and either terminate or continue running. A checkpointed job may be restarted at any time and resume execution from its most recent checkpoint.8.4.2.4 RM Preemption Constraints
Maui is only able to utilize preemption if the underlying resource manager/OS combination supports this capability. The following table displays current preemption limitations:Table 8.4.2.4 Resource Manager Preemption Constraints
| Resource Manager |
OpenPBS (2.3) |
PBSPro (5.2) |
Loadleveler (3.1) |
LSF (5.2) |
SGE (5.3) |
| Cancel |
yes |
yes |
yes |
yes |
??? |
| Requeue |
yes |
yes |
yes |
yes |
??? |
| Suspend |
yes |
yes |
yes |
yes |
??? |
| Checkpoint |
(yes on IRIX) |
(yes on IRIX) |
yes |
(OS dependent) |
??? |
See Also: N/A .
QOS Overview
Managing QOS Access