Moab-BProc Integration Guide
OverviewMoab can be used as an external scheduler for BProc. In this configuration, Moab will keep track of where jobs are running and therefore which nodes are available. This interaction is enabled by a number of scripts which take advantage of Moab's Native Resource Manager. It is important to note that in this configuration Moab must be run with root privileges in order to start each job as the user that submitted it. Because of this, these scripts are run as root as must be appropriately safeguarded against tampering.
Moab ConfigurationMoab drives BProc via perl scripts which interact with the cluster, mostly using the bpsh command. Moab must be given the location of these scripts in the moab.cfg file. An example entry in moab.cfg is:
RMCFG[bproc] TYPE=NATIVE FLAGS=FULLCP RMCFG[bproc] CLUSTERQUERYURL=exec:///$TOOLSDIR/node.query.bproc.pl RMCFG[bproc] WORKLOADQUERYURL=exec:///$TOOLSDIR/job.query.bproc.pl RMCFG[bproc] JOBSUBMITURL=exec:///$TOOLSDIR/job.submit.bproc.pl RMCFG[bproc] JOBCANCELURL=exec:///$TOOLSDIR/job.cancel.bproc.pl RMCFG[bproc] JOBSTARTURL=exec:///$TOOLSDIR/job.start.bproc.pl RMCFG[bproc] JOBSUSPENDURL=exec:///$TOOLSDIR/job.suspend.bproc.pl RMCFG[bproc] JOBRESUMEURL=exec:///$TOOLSDIR/job.resume.bproc.plThese lines configure Moab to use the Native resource manager. FLAGS=FULLCP indicates that Moab will fully checkpoint each job if it is shutdown. This is necessary because Moab itself is keeping track of each job. In each line, "exec:" indicates the source is a program as opposed to a file. $HOME refers to the Moab home directory, not the user home. According to this configuration, the scripts should be located in a directory called "tools" in the Moab home directory.
This script monitors node status. Moab itself keeps track of which nodes are busy, the function of this script is to determine which nodes are up or down from the output of "bpstat -l". This script also attempts to determine node specs via the commands free and uname, and the output of "cat /proc/cpuinfo".
When Moab is ready to run a job, it is first submitted to this script, which outputs the job specifications to a file /tmp/.idlejobs.
After the job specifications have been submitted to job.submit.bproc.pl and written to /tmp/.idlejobs the job is sent to this script which extracts that information, starts the job, and outputs important tracking information such as the process ID to /tmp/.runningjobs. The steps of this interaction are as follows:
This script is used to monitor the status of each job recorded in /tmp/.runningjobs. Additionally, the script will report any unrecorded processes to Moab using the program's process ID as the job id. The script operates as follows:
Sends a kill signal (kill -9) to the indicated job.
Sends a stop signal (kill -19) to the indicated job.
Sends a continue signal (kill -18) to the indicated job.
|© 2001-2010 Adaptive Computing Enterprises, Inc.|