Job Dependencies
Moab Workload Manager®

11.5 Job Dependencies

11.5.1 Basic Job Dependency Support

By default, basic single step job dependencies are supported through completed/failed step evaluation. Basic dependency support does not require special configuration and is activated by default. Dependent jobs are only supported through a resource manager and therefore submission methods depend upon the specific resource manager being used. For TORQUE's qsub and the Moab msub command, the semantics listed in the section below can be used with the -W x=depend:<STRING> flag. For other resource managers, consult the resource manager specific documentation.

NOTE: Situations can arise where idle job limits are set and the dependee is blocked out. To avoid this, use the BLOCKLIST DEPEND parameter.

11.5.1.1 Job Dependency Syntax

DependencyFormatDescription
after after:<jobid>[:<jobid>]... Job may start at any time after specified jobs have started execution.
afterany afterany:<jobid>[:<jobid>]... Job may start at any time after all specified jobs have completed regardless of completion status.
afterok afterok:<jobid>[:<jobid>]... Job may be start at any time after all specified jobs have successfully completed.
afternotok afternotok:<jobid>[:<jobid>]... Job may start at any time after any specified jobs have completed unsuccessfully.
before before:<jobid>[:<jobid>]... Job may start at any time before specified jobs have started execution.
beforeany beforeany:<jobid>[:<jobid>]... Job may start at any time before all specified jobs have completed regardless of completion status.
beforeok beforeok:<jobid>[:<jobid>]... Job may start at any time before all specified jobs have successfully completed.
beforenotok beforenotok:<jobid>[:<jobid>]... Job may start at any time before any specified jobs have completed unsuccessfully.
on on:<count> Job may start after <count> dependencies on other jobs have been satisfied.
synccount synccount:<count> Job must start at the same time as <count> other jobs that reference this job using the syncwith keyword.
syncwith syncwith:<jobid> Job must start at the same time as <jobid>.

11.5.2 Task Scheduling of Directed-Acyclical Graphs (DAGs)

Moab 5.1.0 and later supports task scheduling, which is is enabled using the advanced DAG syntax.

11.5.3 Task Dependency (DAG) Configuration Example

moab.cfg
SCHEDCFG[data1] SERVER=data1.com MODE=normal
ADMINCFG[1]     USERS=root,charlie2,styke

# interface to TORQUE resource manager
RMCFG[torque]   TYPE=PBS

# interface to data store - note will return variables and database 
# health/availability
RMCFG[database] TYPE=native
RMCFG[database] CLUSTERQUERYURL=exec://$HOME/tools/data.import.pl
RMCFG[database] INFOQUERYURL=exec://$HOME/tools/data.register.pl

# daily standing trigger - will submit single daily system job
SCHEDCFG[data1] TRIGGER=AType=exec,Action="/opt/moab/triggers/daily.pl",EType=standing,RearmTime=1:00:00:00,Flags=user+data

# define daily system job
JOBCFG[daily]  USER=data WCLIMIT=48:00:00 GRES=core:8

# launch if hourly jobs do not complete within 15 minutes of target
JOBCFG[daily]  TRIGGER=name=HourInitialize,atype=exec,action="$HOME/tools/daily_init.pl $HOUR $Data3",sets=HourInit.DATASIZE,requires=HOUR.Data3,etype=start
JOBCFG[daily]  TRIGGER=name=HourLaunch,atype=exec,action="$HOME/tools/daily_job_launch.pl $DATASIZE $HOUR",requires=HOUR.DATASIZE.HourInit,sets=!Failed.Success,etype=start  # submit daily
JOBCFG[daily]  TRIGGER=t3  # handle failures
JOBCFG[daily]  TRIGGER=name=HourFail,atype=exec,action="$HOME/tools/daily_job_failure.pl",requires=Failed,etype=start  # handle failures
JOBCFG[daily]  TRIGGER=name=HourTimeout,atype=exec,action="$HOME/tools/daily_job_failure.pl",requires=HourInit.!Failed.!Success,offset=15:00,etype=start # handle failures
# export global values, finalize processing, and cancel job
JOBCFG[daily]  TRIGGER=name=HourFinalize,atype=exec,action="$HOME/tools/daily_finalize.pl",requires=Success.HOUR  # finalize and terminate job

# define hourly system job

JOBCFG[hourly]  USER=data WCLIMIT=2:00:00 GRES=core:2 JGROUP=$$DAY VARIABLE=+hour
# initialize our processing
JOBCFG[hourly]  TRIGGER=name=HourInitialize,atype=exec,action="$HOME/tools/hour_init.pl $HOUR $Data3",sets=HourInit.DATASIZE,requires=HOUR.Data3,etype=start
JOBCFG[hourly]  TRIGGER=name=HourLaunch,atype=exec,action="$HOME/tools/hour_job_launch.pl $DATASIZE $HOUR",requires=HOUR.DATASIZE.HourInit,sets=!Failed.Success,etype=start  # submit hourly
JOBCFG[hourly]  TRIGGER=name=HourFail,atype=exec,action="$HOME/tools/hour_job_failure.pl",requires=Failed,etype=start  # handle failures
JOBCFG[hourly]  TRIGGER=name=HourTimeout,atype=exec,action="$HOME/tools/hour_job_failure.pl",requires=HourInit.!Failed.!Success,offset=15:00,etype=start # handle failures
JOBCFG[hourly]  TRIGGER=name=HourFinalize,atype=exec,action="$HOME/tools/hour_finalize.pl",requires=Success.HOUR  # finalize and terminate job

# set defaults for data mover compute job

JOBCFG[mover]  GRES=netbw:1000,iobw:50 MEMORY=2000 FEATURES=mover ARCH=opteron 
JOBCFG[mover]  JGROUP=$$HOUR
JOBCFG[mover]  VARIABLE=+mover

# set defaults for aggregator compute job

JOBCFG[aggr]   GRES=netbw:200,iobw:250 MEMORY=1000 FEATURES=aggregator 
JOBCFG[aggr]   JGROUP=$$HOUR
JOBCFG[aggr]   VARIABLE=+aggr

# set defaults for search/extract compute job

JOBCFG[extract] GRES=netbw:200,iobw:250 MEMORY=1000 FEATURES=extract
JOBCFG[extract] JGROUP=$$HOUR
JOBCFG[extract] VARIABLE=+extract

# apply generic resources to global and compute resources

NODECFG[cnode001] GRES=netbw:4200,iobw:1000 
NODECFG[cnode002] GRES=netbw:4200,iobw:1000
NODECFG[cnode003] GRES=netbw:4200,iobw:1000
NODECFG[cnode004] GRES=netbw:4200,iobw:1000
NODECFG[cnode005] GRES=netbw:2600,iobw:800
NODECFG[cnode006] GRES=netbw:2600,iobw:800
NODECFG[cnode007] GRES=netbw:2600,iobw:800
NODECFG[cnode008] GRES=netbw:2600,iobw:800
...

#!/bin/sh
NAME=`echo sleep 6000 | qsub -l nodes=1,walltime=6000"`

echo sleep 30 | qsub -l nodes=1,walltime=30 -W x=VAR:+green\;JGROUP:$NAME
echo sleep 30 | qsub -l nodes=1,walltime=30 -W x=VAR:+green\;JGROUP:$NAME
echo sleep 30 | qsub -l nodes=1,walltime=30 -W x=VAR:+green\;JGROUP:$NAME
echo sleep 30 | qsub -l nodes=1,walltime=30 -W x=VAR:+green\;JGROUP:$NAME
echo sleep 30 | qsub -l nodes=1,walltime=30 -W x=VAR:+green\;JGROUP:$NAME
echo sleep 30 | qsub -l nodes=1,walltime=30 -W x=VAR:+green\;JGROUP:$NAME
echo sleep 30 | qsub -l nodes=1,walltime=30 -W x=VAR:+green\;JGROUP:$NAME
echo sleep 30 | qsub -l nodes=1,walltime=30 -W x=VAR:+green\;JGROUP:$NAME
echo sleep 30 | qsub -l nodes=1,walltime=30 -W x=VAR:+green\;JGROUP:$NAME

echo sleep 600 | qsub -l nodes=1,walltime=600,depend=ge:green\|5 -W x=JGROUP:$NAME

#!/bin/sh
NAME=`echo sleep 6000 | qsub -l nodes=1,walltime=6000"`
NAME2=`echo sleep 60 | qsub -l nodes=1,walltime=60 -W x=JGROUP:$NAME`
echo sleep 30 | qsub -l nodes=1,walltime=30 -W x=JGROUP:$NAME2\;VAR:^green:blue

See Also
© 2001-2008 Cluster Resources, Incorporated