Moab-SGE Integration Notes

Moab-SGE Integration Notes

Copyright © 2008 Cluster Resources, Inc.

This document provides information on the steps to integrate Moab with an existing functional installation of SGE.

Notice

Distribution of this document for commercial purposes in either hard or soft copy form is strictly prohibited without prior written consent from Cluster Resources, Inc.

Overview

Moab's native resource manager interface can be used to manage an SGE resource manager. The integration steps simply involve the creation of a complex variable and a default request definition. The moab tools directory contains a collection of customizable scripts which are used to interact with sge. This directory also contains a configuration file for the sge tools.

Moab Integration Steps

You should follow the regular steps for installing Moab with the following exceptions:

Run Configure with the —with-sge option

When running the configure command, use the —with-sge option to specify the use of the native resource manager interface with the sge resource manager subtype. This will place a line similar to the following in the moab configuration file (moab.cfg):

RMCFG[clustername]      TYPE=NATIVE:sge
    

Example 1. Running configure

$ ./configure --prefix=/opt/moab --with-homedir=/var/moab --with-sge

Customize the sge tools configuration file

You may need to customize the tools/config.sge.pl file to include the correct SGE_ROOT and PATH, and set other configuration parameters.

Example 2. Edit config.sge.pl

# vi /opt/moab/tools/config.sge.pl 

# Set the SGE_ROOT environment variable
$ENV{SGE_ROOT} = "/opt/sge-root";

# Set the PATH to include directories for sge commands -- qhost, etc.
$ENV{PATH} = "$ENV{SGE_ROOT}/bin/lx24-x86:$ENV{PATH}";

# Specify job node match criteria as "node"
$jobNodeMatchCriteria = "node";

SGE Integration Steps

After installing SGE on your cluster and verifying that it is running serial and parallel jobs satisfactorily, you should perform the following steps:

Define a new complex variable named node

Use the qconf -mc command to edit the complex variable list and add a new requestable variable of the of the name job and the type RESTRING.

# qconf -mc 

node              node         RESTRING    ==    YES         NO         NONE  0

Add a default node request definition

This step will set the node complex variable for all jobs to the unassigned state until they are ready to run, at which time the job will be assigned a nodelist directing which nodes it can run on.

Example 3. Edit sge_request

# vi /opt/sge-root/default/common/sge_request 

# Set the job's node variable to the unassigned state until it is ready to start
# at which time it will be reset to the list of nodes it is designated to run on
-l node=unassigned

Populate the node's node variable

This step will set the node complex variable for all exec hosts to their own short hostnames. This will allow jobs to start when their node value matches up with a set of nodes.

Example 4. qconf -rattr exechost complex_values node=$hostname $hostname

# for i in `qconf -sel | sed 's/\..*//'`; do echo $i; qconf -rattr exechost complex_values node=$i $i; done

Shorten the scheduler interval

Use the qconf -msconf command to edit the schedule_interval setting to be less than or equal to one half the time of the Moab RMPOLLINTERVAL (seen with showconfig | grep RMPOLLINTERVAL).

# qconf -msconf 

schedule_interval                 0:0:15

© 2001-2008 Cluster Resources, Incorporated