Resource usage limits constrain the amount of resources a given job may consume. These limits are generally proportional to the resources requested and may include walltime, any standard resource, or any specified generic resource. The parameter RESOURCELIMITPOLICY controls which resources are limited, what limit policy is enforced per resource, and what actions the scheduler should take in the event of a policy violation.
The RESOURCELIMITPOLICY parameter accepts a number of policies, resources, and actions using the format and values defined below.
Example - Notify and then cancel job if requested memory is exceeded
# if job exceeds memory usage, immediately notify owner # if job exceeds memory usage for more than 5 minutes, cancel the job RESOURCELIMITPOLICY MEM:ALWAYS,EXTENDEDVIOLATION:NOTIFY,CANCEL:00:05:00
Example - Checkpoint job on walltime violations
# if job exceeds requested walltime, checkpoint job RESOURCELIMITPOLICY WALLTIME:ALWAYS:CHECKPOINT # when checkpointing, send term signal, followed by kill 1 minute later RMCFG[base] TYPE=PBS CHECKPOINTTIMEOUT=00:01:00 CHECKPOINTSIG=SIGTERM
Example - Cancel jobs that use 5% or less of potential CPU usage for more than 5 minutes
Example - Migrating a job when it blocks other workload
Moab is able to perform different actions for both hard and soft policy violations. In most resource management systems, a mechanism does not exist to allow the user to specify both hard and soft limits. To address this, Moab provides the RESOURCELIMITMULTIPLIER parameter that allows per partition and per resource multiplier factors to be specified to generate the actual hard and soft limits to be used. If the factor is less than one, the soft limit will be lower than the specified value and a Moab action will be taken before the specified limit is reached. If the factor is greater than one, the hard limit will be set higher than the specified limit allowing a buffer space before the hard limit action is taken.
In the following example, job owners will be notified by email when their memory reaches 100% of the target, and the job will be canceled if it reaches 125% of the target. For wallclock usage, the job will be requeued when it reaches 90% of the specified limit if another job is waiting for its resources, and it will be checkpointed when it reaches the full limit.
RESOURCELIMITPOLICY MEM:ALWAYS,ALWAYS:NOTIFY,CANCEL RESOURCELIMITPOLICY WALLTIME:BLOCKEDWORKLOADONLY,ALWAYS:REQUEUE,CHECKPOINT RESOURCELIMITMULTIPLIER MEM:1.25,WALLTIME:0.9
While Moab constrains walltime using the parameter RESOURCELIMITPOLICY like other resources, it also allows walltime exception policies which are not available with other resources. In particular, Moab allows jobs to exceed the requested wallclock limit by an amount specified on a global basis using the JOBMAXOVERRUN parameter or on a per credential basis using the OVERRUN attribute of the *CFG credential parameters.
JOBMAXOVERRUN 00:10:00 CLASSCFG[debug] overrun=00:00:30
Searches Moab documentation only
|© 2001-2010 Adaptive Computing Enterprises, Inc.|