Appendix F: Moab Parameters

ACCOUNTCFG[projectX] MAXJOB=50 QDEF=highprio

Up to 50 jobs submitted under the account ID projectX will be allowed to execute simultaneously and will be assigned the QOS highprio by default.

AGGREGATENODEACTIONS TRUE

Queues node actions together when possible.

AGGREGATENODEACTIONSTIME 120

Sets the AGGREGATENODEACTIONS delay to two minutes..

ALLOWROOTJOBS TRUE

Jobs from the root user can execute.

NODECFG[node01] GRES=calclab:1,powerhouse:1 RCSOFTWARE=calclab:1,powerhouse:1
NODECFG[node02] GRES=calclab:1,powerhouse:1 RCSOFTWARE=calclab:1,powerhouse:1
APPLICATIONLIST calclab,powerhouse

The generic resources 'calclab' and 'powerhouse' will now be recognized and treated as application software.

BFPRIORITYPOLICY  DURATION

Use length of job in determining which backfill job to preempt.

CHECKPOINTEXPIRATIONTIME 1:00:00:00

Expire checkpoint data which has been stale for over one day.

CLIENTTIMEOUT 00:15:00

Moab clients will wait up to 15 minutes for a response from the server before timing out.

DELETESTAGEOUTFILES TRUE

Example of an explicit stageout request

msub x=MSTAGEOUT:ssh://source_node/tmp/file,file:///results_folder
job.cmd

With this parameter set to true, /tmp/file on source_node is deleted after it is copied to the specified destination (file:///results_folder). If the parameter is not set, or if it is set to false, /tmp/file remains on source_node after the job terminates.

DISABLEINTERACTIVEJOBS TRUE

Note: It is possible for users to submit interactive jobs directly to a resource manager, which can bypass the DISABLEINTERACTIVEJOBS parameter. However, some resource managers (such as TORQUE) will check with Moab before allowing an interactive job.

RESWEIGHT  10
DISKWEIGHT 100

If a job requires 12 tasks and 512 MB per task of dedicated local disk space, Moab will increase the job's priority by 10 * 100 * 12 * 512

EMULATIONMODE SLURM

Moab will perform the automated setup steps as if it were interfacing with a slurm resource manager (automatic QOS creation).

ENABLEHIGHTHROUGHPUT TRUE

Moab can now accept hundreds of jobs per second using msub instead of 20-30.

ENABLENEGJOBPRIORITY TRUE

Job priority may range from -INFINITY to MMAX_PRIO.

ENABLEPOSUSERPRIORITY TRUE

Users may now specify positive job priorities and have them take effect (e.g. msub -p 100 job.cmd).

FEATURENODETYPEHEADER xnt

Moab will interpret all node features with the leading string xnt as a nodetype specification - as used by Gold and other allocation managers, and assign the associated value to the node. i.e., xntFast.

FEATUREPARTITIONHEADER xpt

Moab will interpret all node features with the leading string xpt as a partition specification and assign the associated value to the node. i.e., xptGold.

FEATUREPROCSPEEDHEADER xps

Moab will interpret all node features with the leading string xps as a processor speed specification and assign the associated value to the node. i.e., xps950.

FEATURERACKHEADER rack

Moab will interpret all node features with the leading string rack as a rack index specification and assign the associated value to the node. i.e., rack16.

FEATURESLOTHEADER slot

Moab will interpret all node features with the leading string slot as a slot index specification and assign the associated value to the node. i.e., slot16.

FEEDBACKPROGRAM /var/moab/fb.pl

Moab will run the specified program at the completion of each job.

FILEREQUESTISJOBCENTRIC TRUE

Moab will treat file requests as a total request per job.

FILTERCMDFILE FALSE

Running the msub command does not perform the actions detailed earlier.

FORCERSVSUBTYPE TRUE

Moab will require all admin reservations to include a subtype.

FSCAP 10.0

Moab will bound a job's pre-weighted fairshare component by the range +/- 10.0.

FSPOLICY DEDICATEDPES

Moab will track fairshare usage by dedicated process-equivalent seconds.

FSTREEACLPOLICY PARENT

Credentials will be given access to their parent node when applicable.

FSTREEISREQUIRED TRUE

Jobs must have an applicable node in the FSTREE in order to execute.

FSTREEUSERISREQUIRED TRUE

Users must be given explicit access to FSTREE nodes in order to gain access to the FSTREE.

GEVENTCFG[hitemp] ACTION=avoid,record,notify  REARM=00:10:00
GEVENT[nodeerror] SEVERITY=3

If a hitemp event is detected, Moab adjusts the node allocation policy to minimize the allocation of the node. Moab also sends emails to cluster administrators and reports the event in the Moab event log.

GRESCFG[scsi1] TYPE=fastio
GRESCFG[scsi2] TYPE=fastio
GRESCFG[scsi3] TYPE=fastio

The generic resources scsi1, scsi2, and scsi3 are all associated with the generic resource type fastio.

IDCFG[info] SERVER=exec:///opt/moab/dbquery.pl REFRESHPERIOD=hour

Moab will refresh credential info every hour using the specified script.

IGNORECLASSES dque,batch

Moab will ignore jobs from classes dque and batch.

IGNOREJOBS !14221,14223

Moab will ignore jobs all classes except 14221 and 14223.

IGNORENODES !host3,host4

Moab will only process nodes host3 and host4.

IGNOREPREEMPTEEPRIORITY TRUE

All preemptor job can preempt any preemptee jobs.

IGNOREUSERS testuser1,annapolis

Moab will ignore jobs from users testuser1 and annapolis.

#INCLUDE moab.acct

 Moab will process the parameters in moab.acct as well as moab.cfg

JOBACTIONONNODEFAILURE REQUEUE

Moab will requeue active jobs which have allocated nodes which have failed during the execution of the job.

JOBAGGREGATIONTIME 00:00:04
RMPOLLINTERVAL     00:00:30

Moab will wait 4 seconds between scheduling cycles when job events have been received and will wait 30 seconds between scheduling cycles otherwise

JOBCFG[sql] REQFEATURES=sqlnode QOS=service

When the sql job is detected, it will have the specified default qos and node feature attributes set.

JOBCPURGETIME 02:00:00

Moab will maintain detailed job information for two hours after a job has completed.

JOBCTRUNCATENLCP TRUE

JOBCTRUNCATENLCP reduces the amount of memory Moab uses to store completed job information.

JOBEXTENDSTARTWALLTIME TRUE

Submit job with a minimum wallclock limit and a walltime; for example:

echo sleep 500 | msub -A ee -l
nodes=5,minwclimit=5:00,walltime=30:00,partition=g02

JOBMAXHOLDTIME 02:00:00

Moab will keep jobs in any HOLD state for 2 hours before canceling them.

JOBMAXOVERRUN 15:00,1:00:00

Jobs may exceed their wallclock limit by up to 1 hour, but Moab will send an email to the primary administrator when a job exceeds its walltime by 15 minutes.

JOBMAXSTARTTIME 2:00:00

Jobs may attempt to start for up to 2 hours before being canceled by the scheduler

JOBNODEMATCHPOLICY EXACTNODE

In a PBS/Native job with resource specification 'nodes=<x>:ppn=<y>', Moab will allocate exactly <y> task on each of <x> distinct nodes.

JOBPREEMPTMAXACTIVETIME 00:05:00

A job is preemptable for the first 5 minutes of its run time.

JOBPREEMPTMINACTIVETIME 00:05:00

A job must execute for 5 minutes before Moab will consider it eligible for preemption.

JOBPRIOACCRUALPOLICY   RESET

Moab will adjust the job's dynamic priority subcomponents, i.e., QUEUETIME, XFACTOR, and TARGETQUEUETIME, etc. each iteration that the job does not violate any JOBPRIOEXCEPTIONS, if it is found in violation, its queuetime will be reset to 0.

JOBPRIOEXCEPTIONS BATCHHOLD,SYSTEMHOLD,DEPENDS

Jobs will accrue priority in spite of batchholds, systemholds, or unsatisfied dependencies.

JOBPURGETIME 00:05:00

Moab will maintain a job record for 5 minutes after the last update regarding that object received from the resource manager.

JOBREMOVEENVVARLIST PBS_SERVER,TZ

Moab will remove the environment variables PBS_SERVER and TZ before submitting jobs.

JOBRETRYTIME   00:05:00

Moab will try for up to 5 minutes to restart jobs if the job start has failed due to transient errors.

LIMITEDJOBCP TRUE

Moab will only maintain scheduler checkpoint information for jobs with explicitly modified job attributes. Some minor job performance and usage statistics may be lost.

LIMITEDNODECP TRUE

Moab will only maintain scheduler checkpoint information for nodes with explicitly modified job attributes. (some minor node performance and usage statistics may be lost)

LOADALLJOBCP TRUE

Moab will load, at startup, all non-completed jobs from all checkpoint files.

LOGDIR /var/spool/moab

Moab will record its log files directly into the /var/spool/moab directory

LOGFACILITY fRM:fPBS

Moab will log only events involving general resource manager or PBS interface activities.

LOGFILE moab.test.log

Log information will be written to the file moab.test.log located in the directory pointed to by the LOGDIR parameter.

LOGFILEMAXSIZE 50000000

Log files will be rolled when they reach 50 MB in size

LOGFILEROLLDEPTH 5

Moab will maintain and roll the last 5 log files.

LOGLEVEL 4

Moab will write all Moab log messages with a threshold of 4 or lower to the moab.log file.

MAXRSVPERNODE 64

64 is the maximum number of reservations on a single node.

MAXRSVPERNODE 100,7000

100 is the maximum number of reservations on a single node, and 7000 is the maximum number of reservations for global nodes.

MAXSUBMITJOBS 1000

Jobs submitted that exceed the limit of 1,000 within the current iteration are rejected.

# free memory associated with leaky RM API
MEMREFRESHINTERVAL 24:00:00

Moab will perform garbage collection once a day.

RESWEIGHT 10
MEMWEIGHT 1000

Each job's priority will be increased by 10 * 1000 * <request memory>.

MSUBQUERYINTERVAL  60

If a user uses the msub -K command, the client remains open and queries the server every 60 seconds until the job completes.

NODEACCESSPOLICY SINGLEUSER

Moab will allow resources on a node to be used by more than one job provided that the job's are all owned by the same user.

NODEALLOCATIONPOLICY MINRESOURCE

Moab will apply the node allocation policy MINRESOURCE to all jobs by default.

NODEALLOCRESFAILUREPOLICY REQUEUE

Moab will requeue jobs which have allocated nodes fail during execution.

NODEAVAILABILITYPOLICY DEDICATED:PROCS COMBINED:MEM

Moab will ignore resource utilization information in locating available processors for jobs but will use both dedicated and utilized memory information in determining memory availability.

NODECATCREDLIST down=BatchFailure,HardwareFailure,NetworkFailure idle=Idle

Moab will create a down user, group, account, class, and QoS and will associate BatchFailure, HardwareFailure, and NetworkFailure resources with these credentials. Additionally, Moab will assign all Idle resources to matching idle credentials.

NODEBUSYSTATEDELAYTIME 0:30:00

Moab will assume busy nodes are not available for scheduling for at least 30 minutes from the current time. Thus, these nodes will never be allocated to starting jobs. Also, these nodes will only be available for reservations starting more than 30 minutes in the future.

NODECFG[nodeA] MAXJOB=2 SPEED=1.2

Moab will only allow two simultaneous jobs to run on node nodeA and will assign a relative machine speed of 1.2 to this node.

NODEDOWNSTATEDELAYTIME 0:30:00

Moab will assume down, drained, and corrupt nodes are not available for scheduling for at least 30 minutes from the current time. Thus, these nodes will never be allocated to starting jobs. Also, these nodes will only be available for reservations starting more than 30 minutes in the future.

NODEDOWNTIME 10:00

If Moab loses communication with the resource manager for more than ten minutes, it sets the state of all nodes belonging to that resource manager to DOWN.

NODEDRAINSTATEDELAYTIME 0:30:00

Moab will assume down, drained, and corrupt nodes are not available for scheduling for at least 30 minutes from the current time. Thus, these nodes will never be allocated to starting jobs. Also, these nodes will only be available for reservations starting more than 30 minutes in the future.

NODEFAILURERESERVETIME 10:00

Moab will reserve failed nodes for 10:00

NODEIDFORMAT *$R*$S

Moab will extract rack and slot information from the cluster node ids (ie, tg-13s08).

NODELOADPOLICY ADJUSTSTATE

Moab will mark a node busy if its measured load exceeds its maximum cpu load limit.

NODEMAXLOAD 0.75

Moab will adjust the state of all idle and running nodes with a load >= .75 to the state busy.

NODEMEMOVERCOMMITFACTOR .5

Moab will overcommit the memory and swap of the node by a factor of 0.5.

NODEPOLLFREQUENCY 5

Moab will update node manager based information every 5 scheduling iterations.

NODEPURGETIME 00:05:00

Moab will maintain a node record for 5 minutes after the last update regarding that object received from the resource manager.

NODESETPOLICY     ONEOF
NODESETATTRIBUTE  PROCSPEED

Moab will create node sets containing nodes with common processor speeds

NODESETATTRIBUTE PROCSPEED
NODESETDELAY     0:00:00

Moab will create node sets containing nodes with common processor speeds. Moab will allocate resources within the nodeset if possible. If no resources exist within a single nodeset, Moab will attempt to run the job using any available resources.

NODESETISOPTIONAL TRUE

Moab will not block a job from running if its node set cannot be satisfied.

NODESETPOLICY     ONEOF
NODESETATTRIBUTE  FEATURE
NODESETLIST       switchA switchB

Moab will allocate nodes to jobs either using only nodes with the switchA feature or using only nodes with the switchB feature.

NODESETPOLICY     ONEOF
NODESETATTRIBUTE  NETWORK

Moab will create node sets containing nodes with common network interfaces.

NODESETPRIORITYTYPE BESTRESOURCE
NODESETATTRIBUTE    PROCSPEED

Moab will select the resource set containing the fastest nodes available.

NODESETATTRIBUTE PROCSPEED
NODESETTOLERANCE 0.5

Moab will only allocate nodes with up to a 50% procspeed difference.

NODETOJOBATTRMAP  fast,big

Jobs requesting node feature fast or big will be assigned a corresponding job attribute.

NODEUNTRACKEDRESDELAYTIME 0:30:00

Moab will assume untracked generic resources are not available for scheduling for at least 30 minutes from the current time. Thus, these nodes will never be allocated to starting jobs. Also, these nodes will only be available for reservations starting more than 30 minutes in the future.

If NODEUNTRACKEDRESDELAYTIME is enabled and there is an untracked resource preventing a job from running, then the job remains in the idle queue instead of being deferred.

PARALLOCATIONPOLICY  LOADBALANCE

New jobs will be started on the most lightly allocated partition.

PBSACCOUNTINGDIR /var/spool/torque/sched_logs/

Job events will be written to the specified directory (can be consumed by PBS's tracejob command).

PARCFG[oldcluster] MAX.WCLIMIT=12:00:00

Moab will not allow jobs to run on the oldcluster partition which has a wallclock limit in excess of 12 hours.

RESWEIGHT 10
PEWEIGHT  100

Each job's priority will be increased by 10 * 100 * its PE factor.

PREEMPTPOLICY CHECKPOINT

Jobs that are to be preempted will be checkpointed and restarted at a later time.

PREEMPTSEARCHDEPTH 8

Serial job preemptors will only consider the first 8 feasible preemptee jobs when determining the best action to take.

PURGETIME 00:05:00

Moab will maintain a job or node record for 5 minutes after the last update regarding that object received from the resource manager.

QOSCFG[commercial] PRIORITY=1000 MAXJOB=4 MAXPROC=80

Moab will increase the priority of jobs using QOS commercial, and will allow up to 4 simultaneous QOS commercial jobs with up to 80 total allocated processors.

QOSISOPTIONAL TRUE
USERCFG[bob]  QLIST=high,low

Moab will set the QOSList for user "bob" to high and low but will not set the QDEF. Should "bob" decide to submit to a particular QOS he will have to do so manually.

QUEUETIMECAP    10000
QUEUETIMEWEIGHT 10

A job that has been queued for 40 minutes will have its queuetime priority factor calculated as 'Priority = QUEUETIMEWEIGHT * MIN(10000,40)'.

QUEUETIMEWEIGHT 20

A job that has been queued for 4:20:00 will have a queuetime priority factor of 20 * 260.

RECORDEVENTLIST JOBSTART,JOBCANCEL,JOBEND

When a local and/or remote job starts, is canceled, or ends, the respective event will be recorded.

REJECTINFEASIBLEJOBS TRUE
JOBREJECTPOLICY      MAIL,CANCEL

Any job with zero feasible nodes for execution will be rejected.

ENABLENEGJOBPRIORITY TRUE 
REJECTNEGPRIOJOBS    TRUE

Any job with a priority less than zero will be rejected.

REMAPCLASS     batch
REMAPCLASSLIST short,medium,long

Class batch will be re-mapped to one of the listed classes.

RESCAP 1000

The total resource priority factor component of a job will be bound by +/- 1000

RESERVATIONDEPTH[bigmem]   4
RESERVATIONQOSLIST[bigmem] special,fast,joshua

Jobs with QOS's of special, fast, or joshua can have a cumulative total of up to 4 priority reservations.

RESERVATIONPOLICY          CURRENTHIGHEST
RESERVATIONDEPTH[DEFAULT]  2

Moab will maintain reservations for only the two currently highest priority jobs.

RESERVATIONDEPTH[big]   4
RESERVATIONQOSLIST[big] hi,low,med

Jobs with QOS's of hi, low, or med can have a cumulative total of up to 4 priority reservations.

RESERVATIONRETRYTIME   00:05:00

Moab will try for up to 5 minutes to maintain immediate reservations if the reservations are blocked due to node state, network, or batch system based race conditions.

RESOURCELIMITMULTIPLER  PROC:1.1,MEM:2.0

Sets hard limit for PROC at 1.1 times the PROC soft limit, and the hard limit of MEM to 2.0 times the MEM soft limit.

RESOURCELIMITPOLICY  MEM:ALWAYS,BLOCKEDWORKLOADONLY:REQUEUE,CANCEL

Moab will cancel all jobs which exceed their requested memory limits.

RESOURCELIST  node01,node05,node06

Moab will query the native rm for information about nodes node01, node05, and node06.

RESTARTINTERVAL  20:00:00

Moab daemon will automatically restart every 20 hours.

RESOURCEQUERYDEPTH  1

The 'mshow -a' command will return at most one valid collection of resources.

RESWEIGHT  5
MEMWEIGHT  10
PROCWEIGHT 100
SWAPWEIGHT 0
RESCAP     2000

The job priority resource factor will be calculated as MIN(2000,5 * (10 * JobMemory + 100 * JobProc)).

RMCFG[TORQUE3] TYPE=PBS

The PBS server will be used for resource management.

RMPOLLINTERVAL 30,45

Moab will refresh its resource manager information between a minimum of 30 seconds and a maximum of 45 seconds. Note: This parameter specifies the default global poll interval for all resource managers.

RSVPROFILE[fast] USERLIST=john,steve
RSVPROFILE[fast] QOSLIST=high,low
RSVPROFILE[fast] TRIGGER=ETYPE=start,OFFSET=5:00,ATYPE=exec,ACTION="/opt/moab/rp.pl"

Moab will create a reservation profile including trigger and ACL information.

RSVREALLOCPOLICY failure

Should any node go down within a reservation Moab will attempt to allocate a replacement. Moab will NOT allocate a busy or otherwise conflicting node. If Moab should allocate any node for replacement the IGNRSV flag must be used when creating the reservation.

SHOWMIGRATEDJOBSASIDLE TRUE

When set to TRUE, migrated jobs will show as idle and will count against the idle policies of the cluster showing the job as migrated.

SIMAUTOSHUTDOWN TRUE

The simulation will end as soon as there are no jobs running and no idle jobs which could run.

SIMDEFAULTJOBFLAGS DEDICATED

Moab will set the DEDICATED job flag on all jobs loaded from the workload trace file.

SIMFLAGS IGNHOSTLIST,IGNQOS

Moab will ignore hostlist and qos requests specified in the workload trace file.

SIMIGNOREJOBFLAGS DEDICATED

Moab will ignore the DEDICATED job flag if specified in any job trace.

SCHEDCFG[sim1]         MODE=SIMULATION
SIMINITIALQUEUEDEPTH   64
SIMJOBSUBMISSIONPOLICY CONSTANTJOBDEPTH

Moab will initially place 64 idle jobs in the queue and, because of the specified queue policy, will attempt to maintain this many jobs in the idle queue throughout the duration of the simulation.

SIMJOBSUBMISSIONPOLICY NORMAL

Moab will submit jobs with the relative time distribution specified in the workload trace file.

SIMRESOURCETRACEFILE traces/nodes.1

Moab will obtain resource traces when running in simulation mode from the <MOABHOMEDIR>/traces/nodes.1 file.