8.4 Preemption Management
Many sites possess workloads of varying importance. While it may be critical that some jobs obtain resources immediately, other jobs are less turnaround time sensitive but have an insatiable hunger for compute cycles, consuming every available cycle. These latter jobs often have turnaround times on the order of weeks or months. The concept of cycle stealing handles such situations well and enables systems to run low priority, preemptible jobs whenever something more pressing is not running. These other systems are often employed on compute farms of desktops where the jobs must vacate anytime interactive system use is detected.
The mjobctl command can be used to preempt jobs. Specifically, the command can be used to modify a job's execution state in the following ways:
In general, users are allowed to suspend or terminate jobs they own. Administrators are allowed to suspend, terminate, resume, and execute any queued jobs.
Moab's QoS-based preemption system allows a site the ability to specify preemption rules and control access to preemption privileges. These abilities can be used to increase system throughput, improve job response time for specific classes of jobs, or enable various political policies. All policies are enabled by specifying some QoS's with the flag PREEMPTOR, and others with the flag PREEMPTEE. For example, to enable a cycle stealing high throughput cluster, a QoS can be created for high priority jobs and marked with the flag PREEMPTOR; another QoS can be created for low priority jobs and marked with the flag PREEMPTEE.
If desired, the RESERVATIONPOLICY parameter can be set to NEVER. With this configuration, low priority, preemptee jobs can be started whenever idle resources are available. These jobs are allowed to run until a high priority job arrives, at which point the necessary low priority jobs are preempted and the needed resources freed. This allows near immediate resource access for the high priority jobs. Using this approach, a cluster can maintain near 100% system utilization while still delivering excellent turnaround time to the jobs of greatest value.
It is important to note the rules of QoS based preemption. Preemption only occurs when the following 3 conditions are satisfied:
Use of the preemption system need not be limited to controlling low priority jobs. Other uses include optimistic scheduling and development job support.
In the below example, high priority jobs are configured to always be able to preempt low priority jobs but not med or other high priority jobs.
PREEMPTPOLICY REQUEUE # enable qos priority to make preemptors higher priority than preemptees QOSWEIGHT 1 QOSCFG[high] QFLAGS=PREEMPTOR PRIORITY=1000 QOSCFG[med] QOSCFG[low] QFLAGS=PREEMPTEE # associate class 'special' with QOS high CLASSCFG[special] QDEF=high&
As in the previous example, any class can be bound to a particular QoS using the QDEF attribute of the CLASSCFG parameter with the & marker.
In environments where job checkpointing or job suspension incur significant overhead, it may be desirable to constrain the rate at which job preemption is allowed. The parameter JOBPREEMPTMINACTIVETIME can be used to throttle job preemption. In essence, this parameter prevents a newly started or newly resumed job from being eligible for preemption until it has executed for the specified time frame. Conversely, jobs can be excluded from preemption after running for a certain amount of time using the JOBPREEMPTMAXACTIVETIME parameter.
The PREEMPT backfill policy allows a site to take advantage of optimistic scheduling. By default, backfill only allows jobs to run if they are guaranteed to have adequate time to run to completion. However, statistically, most jobs do not use their full requested wallclock limit. The PREEMPT backfill policy allows the scheduler to start backfill jobs even if required walltime is not available. If the job runs too long and interferes with another job that was guaranteed a particular timeslot, the backfill job is preempted and the priority job is allowed to run. When another potential timeslot becomes available, the preempted backfill job will again be optimistically executed. In environments with checkpointing or with poor wallclock accuracies, this algorithm has potential for significant savings. See the backfill section for more information.
Rules regarding which jobs can be preemptors and which are preemptees can be configured to take into account aspects of the compute environment. Some of these context sensitive rules are listed here:
How the scheduler preempts a job is controlled by the PREEMPTPOLICY parameter. This parameter allows preemption to be enforced using one of the following methods: suspend, checkpoint, requeue, or cancel.
Under this policy, active jobs are terminated and returned to the job queue in an idle state.
Suspend causes active jobs to stop executing but to remain in memory on the allocated compute nodes. While a suspended job frees up processor resources, it may continue to consume swap and other resources. Suspended jobs must be resumed to continue executing.
Systems that support job checkpointing allow a job to save off its current state and either terminate or continue running. A checkpointed job may be restarted at any time and resume execution from its most recent checkpoint.
See Checkpoint/Restart Facilities for more information.
Under this policy, active jobs are canceled.
Moab is only able to use preemption if the underlying resource manager/OS combination supports this capability. The following table displays current preemption limitations:
Table 220.127.116.11 Resource Manager Preemption Constraints
There are multiple steps associated with setting up a working preemption policy. With preemption, issues arise because it appears that Moab is not allowing preemptors to preempt preemptees in the right way. To diagnose this, use the following checklist:
Searches Moab documentation only
|© 2001-2010 Adaptive Computing Enterprises, Inc.|