Moab-XT3 Integration Guide

Moab-XT3 Integration Guide

Overview

   Moab can be used to manage the batch system for a Cray XT3 supercomputer.  Moab can run with either Torque or PBS Pro as the underlying resource manager allowing end-users to submit, monitor, and control jobs using familiar interfaces and tools.  However, because the Moab/Torque solution offeres a much larger set of capabilities including robust reservations, heterogenous node support, improved scheduling, node features, and improved cpa allocation failure recovery, this document will focus on the Moab and Torque integrations steps.

1.0 Installing Moab on the Cray XT3

NOTE: The following steps assume:
  • that the Cray XT3 System has been configured to have a persistent /var and
  • that all userids are resolvable from the sdb host.

1.1 Install Torque on your system

If Torque is not already installed on your system, use the instructions found here to complete the installation.

1.2 Download the latest moab snapshot from Cluster Resources, Inc.

On the boot node as root:

download moab
> cd /tmp
> wget --http-user=<cri_download_user> --http-passwd=<cri_download_password> http://www.clusterresources.com/downloads/mwm/temp/moab-5.1.0p4-snap.200705111202.tar.gz

1.3 Unpack the moab tarball into the software directory in the shared root

unpack moab
> cd /rr/current/software
> tar -zxvf /tmp/moab-5.1.0p4-snap.200705111202.tar.gz

1.4 Use xtopview to configure and install torque into the shared root

xtopview
>  xtopview
default/:/ # cd /software/moab-5.1.0p4-snap.200705111202

1.5 Configure Moab

While still in xtopview, run configure with the options set appropriately for your installation. CRI recommends installing the moab binaries into /opt/moab/$version and establishing a symbolic link to it from /opt/moab/default. Since the moab home directory must be read-write by root, CRI recommends you specify the homedir in a location such as /var/spool/moab.

configure moab
default/:/software/moab-5.1.0p4-snap.200705111202 # ./configure --homedir=/var/spool/moab --instdir=/opt/moab/5.1.0p4-snap.200705111202

1.6 Install moab

While still in xtopview, install moab into the shared root. You may also need to link /opt/moab/default to this installation. Exit xtopview.

install moab
default/:/software/5.1.0p4-snap.200705111202 # mkdir -p /var/spool/moab
default/:/software/5.1.0p4-snap.200705111202 # make install
default/:/software/5.1.0p4-snap.200705111202 # ln –sf /opt/moab/5.1.0p4-snap.200705111202/ /opt/moab/default
default/:/software/5.1.0p4-snap.200705111202 # exit

1.7 Copy your moab home directory to your moab server host

In this example we assume the moab server will be running on the sdb node. If you are installing moab with its server home in /var as in this example and assuming that your var filesystem is being served from your boot node under /snv, you will need to login to sdb and determining the nid with ‘cat /proc/cray_xt/nid’.

copy home directory
> cd /rr/current/var/spool
> cp –pr moab /snv/3/var/spool

1.8 Customize the moab configuration file for your moab server host

customize moab configuration
> cd /snv/3/var/spool/moab
> vi moab.cfg
SCHEDCFG[moab]       SERVER=sdb:42559
RMCFG[mycluster]  TYPE=native:xt3
NODECFG[DEFAULT] OS=linux ARCH=xt

1.9 Copy the moab configuration file to all of the login nodes

The only essential parameter is the SCHEDCFG line so the clients can find the server. This example assumes you are using a persistent /var filesystems mounted from /snv on the boot node and that your login nodes have nids of 4, 64 and 68. Alternatively, a ram var filesystem must be populated by a skeleton tarball on the bootnode (/rr/current/.shared/var-skel.tgz) into which these files must be added.

example
> for i in 4 64 68
> do mkdir -p /snv/$i/var/spool/moab
> cp moab.cfg /snv/$i/var/spool/moab
> done

1.10 Customize the xt3 native resource manager interface configuration and scripts

These are located in the $prefix/tools directory by default and consist of a configuration file (config.xt3.pl) and various scripts (job.query.xt3.pl, node.query.xt3.pl, job.start.xt3.pl, job.cancel.xt3.pl, …).

customize moab tools
> cd /rr/current/opt/moab/default/tools
> vi config.xt3.pl
  "$ENV{PATH}:/opt/torque/default/bin:/opt/MySQL/default/bin:$FindBin::Bin";

1.11 Compile the cpaquery application

NOTE: This is a temporary requirement and will be removed in a later release.

compile cpaquery
> gcc cpaquery.c -I/opt/xt-pe/default/include -L/opt/xt-os/default/lib/snos64 -l cpalib -o cpaquery

1.12 Copy in the init.d script (Optional)

Moab provides an init.d script for starting Moab as a service.

Copy in init.d script
> cd /rr/current/software/moab-5.2.0.s6854
> cp contrib/init.d/moab /etc/init.d
> chmod +x /etc/init.d/moab
# Edit file as necessary -- i.e. change PATH to use /opt/moab/default
> vi /etc/init.d/moab
path=/opt/moab/default/sbin

1.13 Copy in the module files (Optional)

Moab provides module files that can be used to establish the proper Moab and Torque environments.

Copy in module files
> cd /rr/current/software/moab-5.2.0.s6854
> mkdir /etc/modulefiles/moab
> cp contrib/modulefiles/moab /etc/modulefiles/moab/.module
> cd /etc/modulefiles/moab
> ln -s .module moab-5.2.0.s6854
> cat >.version <<EOF
#%Module1.0
set ModulesVersion      "moab-5.2.0.s6854"
EOF

1.14 Start up the moab workload manager

On the moab server host as root:

start moab
> /opt/moab/default/sbin/moab
© 2001-2008 Cluster Resources, Incorporated