Moab Workload Manager®

19.4 Trigger Types

• 19.4.1 Mail Triggers
• 19.4.2 Internal Triggers
• 19.4.3 Threshold Triggers
• 19.4.4 Exec Triggers

19.4.1 Mail Triggers

When the mail option is used for the AType parameter, Moab will send mail (configurable using the MAILPROGRAM) to the primary administrator. When AType=mail, the Action parameter contains the message body of the email. This can be configured to include certain variables. The following example sends an email to the primary administrator containing the name of the trigger's object, and the time:

>mrsvctl -c -h node01 -T AType=mail,EType=start \
  Action="rsv $OID started on $TIME on nodes $HOSTLIST"

rsv 'system.1' created

When this trigger launches, it will send an email with the variables (that begin with a $) filled in. Possible variables that can be inserted into the Action parameter include the following:

• OID is the name of the object to which the trigger was attached.
• TIME is the time of the trigger launch.
• HOSTLIST is the HostList of the trigger's object (when applicable).
• OWNER is the owner of the trigger's object (when applicable).
• USER is the user (when applicable).
• Any other variable documented in the Default Variables section.

These mail triggers can be configured to launch for node failures, reservation creation or release, scheduler failures, and even job events. In this way, site administrators can keep track of scheduler events through email.

19.4.2 Internal Triggers

Triggers can be used to modify object internals using the following format:

action="<OBJECT_TYPE>:<OBJECT_ID>:<ACTION>:<CONTEXT_DATA>

Several different actions are valid depending on what type of object the internal trigger is acting upon. The following table shows some of the different available actions:

Examples — Modifying scheduler internals

SRCFG[provision] TRIGGER=atype=internal,etype=start,action="node:$(HOSTLIST):modify:os=rhel4"

$ msub -l  
qos=triggerok,walltime=60,trig=AType=internal\&Action=vpc:vpc.20:destroy\&EType=end testscript

19.4.3 Threshold Triggers

Threshold triggers allow sites to configure triggers to launch based on internal scheduler statistics, such as the percentage of available nodes, the backlog of a particular QoS, or the xfactor of a particular group. For example, a site may need to guarantee a particular account a certain level of service on the cluster. Should the specified account have a backlog of over one hour, the administrators would like to receive an email, or create a reservation, or perhaps contact a hosting utility to allocate more nodes. All of this can be done using threshold triggers. The following sample configuration file contains a few examples:

RMCFG[TORQUE]   TRIGGER=atype=internal,action=RM:ODM:modify:nodes:1,etype=start,threshold=availability<300,multifire=true,failoffset=1:00
CLASSCFG[batch] TRIGGER=atype=mail,action="batch has a lot of jobs waiting",etype=start,threshold=queuetime>100000,multifire=true
QOSCFG[high]    TRIGGER=atype=exec,action=/tmp/reserve_high.sh,etype=start,threshold=backlog>1000,multifire=true,failoffset=10:00
USERCFG[jdoe]   TRIGGER=atype=mail,action="high xfactor on $OID",etype=start,threshold=xfactor>10.0,multifire=true,failoffset=1:00
ACCTCFG[hyper]  TRIGGER=atype=exec,action="/tmp/notify.sh $TIME $OID",etype=start,threshold=xfactor>0.01,multifire=true,failoffset=1:00

19.4.4 Exec Triggers

Exec triggers will launch an external program or script when their dependencies are fulfilled. The following example will submit job.cmd and then execute monitor.pl three minutes after the job is started.

> msub -l trig=atype=exec\&etype=start\&offset=03:00\&action="monitor.pl" job.cmd

See Also

• Event Handling Case Study
• Generic Metrics
• Generic Events