[Mauiusers] My SLURM/Maui configuration notes

[Balle, Susanne]

Hi,

I know that I am not the only one struggling with the SLURM/Maui
integration so I want to share my findings with whoever else is trying
to make that combination of job scheduler/resource manager work. 

I am hoping that we can get an active group/discussion going so that we
can get the SLURM/Maui combination tested more intensively and
extensively and hopefully gets some of the bugs fixed in either Maui or
SLURM fixed and/or better understand the limitation of Maui/SLURM.

I have enclosed my SLURM/Maui configuration notes below. I am working
with SLURM and Maui on an HP XC cluster so some of the tips might be XC
specific but I believe that most should work for any system. 

Regards

Susanne

---------------------------------------------------------------
Susanne M. Balle, PhD
Hewlett-Packard

Susanne.Balle at hp.com


SLURM/Maui Configuration
------------------------
The SLURM website has the following webpage with instructions on how to
integrate SLURM and Maui: http://www.llnl.gov/linux/slurm/maui.html.
Unfortunately these instructions are out of date but the general
guidelines are still correct with the exception of several omissions.

In this Section, we go over step by step how we built the Maui scheduler
and make it interoperate with SLURM. 

Step 1: Download the Maui scheduler kit: maui-3.2.6p9 from the Maui
website: http://www.clusterresources.com/products/maui/

We used the version of SLURM that comes with XC namely slurm-0.3.8.
SLURM was installed and running on the XC before we installed Maui.

Step 2: Compile MAUI from its source distribution. This is a two step
process:
1.	./configure --with-key=42 --with-wiki
2.	gmake

Step 3: Update the Maui configuration file: maui.cfg. (This file is
located in <path the Maui  directory>/maui-3.2.6p9/maui.cfg.dist. Copy
the maui.cfg.dist to maui.cfg)

In the Maui configuration file (maui.cfg), add the following
configuration parameters (in bold below):

RMCFG[XC14N16] TYPE=WIKI
RMPORT          7321
RMHOST          XC14N16
RMAUTHTYPE[XC14N16] NONE

Note: XC14N16 is the hostname where the SLURM controller is running.
This has to match the ControlMachine in
/hptc_cluster/slurm/etc/slurm.conf.

The SLURM integration guide suggests having Maui poll SLURM often - in
this case every 20 seconds since a job submitted to an idle cluster will
not be initiated until the Maui daemon polls SLURM and decides to make
it run. 

RMPOLLINTERVAL  00:00:20

In order for Maui to be able to access your SLURM partition you will
need to define a partition with the same name as your SLURM partition in
the maui.cfg file.

>From /hpcd_cluster/slurm/etc/slurm.conf: 
PartitionName=lsf Default=YES Shared=Yes Nodes=xc14n[13-16]

Update maui.cfg to include:

PARTITIONMODE ON

and

NODECFG[xc14n16]	PARTITION=lsf 
NODECFG[xc14n15]	PARTITION=lsf 
NODECFG[xc14n14]	PARTITION=lsf 
NODECFG[xc14n13]	PARTITION=lsf 

Step 4: Uncomment the following lines in
/hptc_cluster/slurm/etc/slurm.conf 
SchedulerType=sched/wiki
SchedulerAuth=42
SchedulerPort=7321

And change AuthType=auth/munge (default XC slurm configuration) to
AuthType=auth/none

Step 5: Set the following environment variable and PATH 
*	set path=(/root/MAUI/maui-3.2.6p9/bin $path)
*	setenv MAUIHOMEDIR /root/MAUI/maui-3.2.6p9

The Maui Administrator's Guide has a section on Building and Installing
Maui which is very useful. The guide can be found at:
http://www.clusterresources.com/products/maui/docs/mauiadmin.shtml