[gold-users] gold fails to remove reservations

[Eva Hocks]

Scott,

thanks for your help!

maui/gold charges in hours due to
ChargeRate Modify Type==Resource Name==Processors Rate=0.00027777

The checkjob -v had
job is deferred.  Reason:  BankFailure  (cannot debit job account)
Holds:    Defer  (hold reason:  BankFailure)

We do not use any prolog/epilog with maui, just with torque.

What I found out though maui added 10 minutes to the end time of the job due to:
RESOURCELIMITPOLICY MEM:ALWAYS,EXTENDEDVIOLATION:NOTIFY,CANCEL:00:05:00
RESOURCELIMITPOLICY WALLTIME:ALWAYS:NOTIFY,CANCEL:00:05:00

which may have gotten the debit for the requested time greater that the credit
(maui seems to round up to the next hour while moab only did so for more than 30
minutes)

I solved the problem with removing the reservations in gold
goldsh Reservation Delete Id==953163

and added some more credits and the job started.


Thanks again for your help.
Eva



On Thu, 24 May 2012, Scott Jackson wrote:

> Eva,
>
> On Wed, May 23, 2012 at 2:01 PM, Eva Hocks <hocks at sdsc.edu> wrote:
>
> >
> > using maui3.2.6p21-1/gold 2.2.0.1-1: a job requesting 10 hours on 96
> > processors
> > doesn't start due to cannot debit account but it did indeed have sufficient
> > credit.
> >
> > $ showq|grep 964040
> > 964040               xyz   Deferred    96    10:00:00  Wed May 23 10:55:02
> >
> > job asking for 96 processors for 10 hours * 1 = 960
> >
>
> First, If this is a 10 hour job, I am little confused as to why maui is not
> charging 96*10*3600. As far as I knew, maui charges by second and not by
> hour.
>
> What does a checkjob -v on the job say is the reason for the deferral?
>
>
> > $ gbalance -u fpaolo
> > Id  Name   Amount Reserved Balance CreditLimit Available
> > --- ------ ------ -------- ------- ----------- ---------
> > 921 xyz    1965     1920      45           0        45
> >
> >
> >
> > looking at the gold database why does one job 964040 have 2 reservations ?
> >
> > $ goldsh Reservation Query Name==964040
> > Id     Name   Job     User   Project Machine StartTime           EndTime
> >           Description
> > ------ ------ ------- ------ ------- ------- -------------------
> > ------------------- -----------
> > 953044 964040 1011912 fpaolo fpaolo  Triton  2012-05-23 10:55:03
> > 2012-05-23 21:05:03
> > 953239 964040 1012109 fpaolo fpaolo  Triton  2012-05-23 12:36:33
> > 2012-05-23 22:46:33
> >
> >
> I am not sure, but it would be a problem in maui for leaving the
> reservations around (or a problem in the prolog and epilog scripts since I
> don't see how maui is going to be charging 960 for the job). One
> improvement that we made in Moab was in linking the jobs up better. When
> Maui was written, if a job tried to start multiple times, it could easily
> wind up creating multiple reservations. Later we added some flags to Gold
> to better address this situation (Moab calls Job Reserve with the
> Replace:=True flag which deletes any reservations of the same name before
> creating a new reservation). I suspect that either this is the problem (and
> you are using an edited version of maui for the hours thing) or that you
> are using prologs and epilogs which do not call the Replace option.
>
> Scott
>
>
> >
> > Id      Object Action  Actor Name Child JobId  Amount Delta Account
> > Project User   Machine Allocation Count Description Details
> >
> >
> >                                                   CreationTime
> >  ModificationTime    Deleted RequestId TransactionId ------- ------ -------
> > ----- ---- ----- ------ ------ ----- ------- ------- ------ -------
> > ---------- ----- ----------- ---
> > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> > ------------------- ------------------- ------- --------- -------------
> > 9462919 Job    Reserve maui             964040    960               fpaolo
> >  fpaolo Triton             1
> > WallDuration=36000,Processors=96,QualityOfService=DEFAULT,Queue=batch,Stage=Reserve,Charge=0,CallType=Normal,ItemizedCharges:=(
> > ( ( 96 [Processors] * 0.00027777 [ChargeRate{VBR}{Processors}] ) ) * 36000
> > [WallDuration] ) = 960 2012-05-23 10:55:06 2012-05-23 10:55:06 False
> > 2901179   9462919
> > 9465533 Job    Reserve maui             964040    960               fpaolo
> >  fpaolo Triton             1
> > WallDuration=36000,Processors=96,QualityOfService=DEFAULT,Queue=batch,Stage=Reserve,Charge=0,CallType=Normal,ItemizedCharges:=(
> > ( ( 96 [Processors] * 0.00027777 [ChargeRate{VBR}{Processors}] ) ) * 36000
> > [WallDuration] ) = 960 2012-05-23 12:36:44 2012-05-23 12:36:44 False
> > 2901826   9465533
> >
> > It does not delete those correctly thus any further try at statring fails
> > as
> > well. Any idea why the reservation request is not removed? It does not
> > happen
> > with other jobs.
> >
> > Thanks
> > Eva
> >
> >
> >
> >
> >
> >
> >
> >
> > _______________________________________________
> > gold-users mailing list
> > gold-users at supercluster.org
> > http://www.supercluster.org/mailman/listinfo/gold-users
> >
>