[Mauiusers] Problem with Standing Reservations when having a large number of jobs running

Michel Jouvin jouvin at lal.in2p3.fr
Thu May 29 14:14:08 MDT 2008


Hi,

I already sent a message to the list one month ago about my problems using 
standing reservations in a cluster with 200 machines/ 1500 cores but it 
remained unanswered.

We are running :

torque-devel-2.3.0-snap.200801151629.2
maui-client-3.2.6p20-snap.1182974819.9

rebuilt with some non default hard limits, in particular for reservations 
and standing reservations :

Parameter : Default Setting : Current Setting :
MAX_MCLASS : 16 : 64 :
MMAX_JOB : 4096 : 32768 :
MAX_MJOB_TRACE : 4096 : 32768 :
MAX_MRES : 1024 : 8192 :
MMAX_SRES : 128 : 1024
MMAX_NODE : 5129 : 5120 :

Our configuration was including 1 standing reservation per machine (around 
250 in total). As long as there is a small number of jobs running 
everything is ok : diagnose -r displays allow the active reservations and 
standing reservations.

After reaching a certain number of running jobs (almost all our jobs are 
using 1 core) we have not been able to determine precisely (between 900 and 
1200, doesn't seem to be 1024!), diagnose -r is no longer able to list all 
reservations and output is ending with a message like :

------
NOTE:  list truncated

Active Reserved Processors: 1235
WARNING:  reservation table is corrupt:  active procs reserved does not 
equal active procs detected (1235 != 2097)
-----

where 2097 is slightly less than the number of running jobs (whatever is 
the actual number, it is always close to number of running jobs - 20). When 
this problem begins, standing reservations are no longer there and the 
consequence is that Torque PROCS normally reserved by standing reservations 
appear free and jobs are scheduled using these PROCS leading to an 
unexpected load on worker nodes. Unfortunatly there is no message in MAUI 
log file... but the impact on scheduling shows this is not a diagnose 
problem.

Even if this is not clear if the problem is related, under these high load 
conditions MAUI is crashing very frequently (we have a cron job that 
restarts it every 5 minutes if it is no longer there).

Thanks in advance for any help, hint or troubleshooting advice. BTW, is 
there any more recent version of MAUI than the one we are running ? I have 
not found anything on clusterressources.com web site but there is may be a 
CVS or SVN repository where to download more recent snapshots.

Cheers,

Michel

     *************************************************************
     * Michel Jouvin                 Email : jouvin at lal.in2p3.fr *
     * LAL / CNRS                    Tel : +33 1 64468932        *
     * B.P. 34                       Fax : +33 1 69079404        *
     * 91898 Orsay Cedex                                         *
     * France                                                    *
     *************************************************************




More information about the mauiusers mailing list