[gold-users] Project doesn't get marked active

Scott Jackson scottmo at adaptivecomputing.com
Mon Nov 29 11:54:20 MST 2010


Victor,

I would recommend not disabling the project unless there is an exception that should prevent them from being able to use their allocated credits. I would think that normally you would use the allocation start and end times to manage when the user/project can use their allocation. I look at the ability to disable a particular user or project as a way to disable them in an exceptional case where you want to preserve the original allocation limits, but just want to make a temporary or partial permanent exception. It is also a fast toggle. You could switch a set of projects off for an emergency workload, then switch them back on, all without messing with their allocation times.

I don't know your charging pattern, but I am sure you are well aware of the recommended mechanism -- to do a sanity check through Job Quote and Account Balance at submit time, a Job Reserve at start time, and a Job Charge at end time. This is the best pattern I've found to prevent the problem conditions. Even this is not perfect -- but I have not thought of anything better than this to date. Many other models are possible with Gold, you just have to make the right calls at the desired times.

Thanks,

Scott


----- Original Message -----
> From: "Victor Gene Hazlewood" <vhazlewo at utk.edu>
> To: "Scott Jackson" <scottmo at adaptivecomputing.com>
> Cc: "Gold Users Mailing List" <gold-users at supercluster.org>
> Sent: Wednesday, November 24, 2010 6:05:48 AM
> Subject: RE: [gold-users] Project doesn't get marked active
> Yes, good explanation. I didn't think about the broader context of the
> cases you pointed out.
> 
> What this means for me as I program all the possible cases is that if
> a project and allocation that do correspond are both expired/inactive
> and then a request comes in to change the end date of an existing
> allocation to make it active currently, I must also set the project as
> being active. In our use of Gold this situation happens once in a
> while when the Teragrid sends a request to change the end date of an
> expired allocation that has time left. We do not integrate Torque and
> Gold (mainly for performance reasons) directly, there can be long
> delays between when jobs are submitted and when they actually
> complete, and the project/allocation checking for a job submission and
> checking at job posting (gcharge) are not done exactly the same way.
> All of these things lead to complexity which requires lots of case
> checking for all the cases in multiple places where these are
> automated.
> 
> Thanks for the explanation.
> 
> -Victor
> 
> -----Original Message-----
> From: Scott Jackson [mailto:scottmo at adaptivecomputing.com]
> Sent: Tuesday, November 23, 2010 9:22 PM
> To: Hazlewood, Victor Gene
> Cc: Gold Users Mailing List
> Subject: Re: [gold-users] Project doesn't get marked active
> 
> Victor,
> 
> I believe this is acting the way I would want it to. The Active status
> of a project is completely separate from the active status of an
> allocation. Projects and Allocations do not have a one-to-one mapping.
> An administrator may set a project as inactive even if one or more of
> the allocations that that project might use are active -- and
> vice-versa, an administrator might mark a project as active, but there
> may be no active allocations remaining. The activeness of the
> allocation kind of happens naturally via the flow of time whereas the
> activeness of a project is an administrative decision by the admin.
> 
> An example of how they might not be one-to-one is if there is a single
> allocation usable by all projects, users and machines (ANY, ANY, ANY).
> This allocation could continue to remain alive while a particular
> project could be disabled by setting its Active state to False. You
> can likewise have a project with multiple allocations. Activity of an
> allocation is defined as being whether the current time falls within
> its start and end time. This cannot be administratively managed
> without messing with the time frames. Generally I would think you
> would not want to do that. You can disable an individual machine, user
> or project without messing with the associated allocations.
> 
> When an allocation expires, it does not mark the project inactive,
> only that allocation. There can be more than one allocation for a
> given project (different timeframes, different collections of users,
> different machines, shared by multiple projects, etc).
> 
> Does that help?
> 
> Thanks,
> 
> Scott
> 
> 
> ----- Original Message -----
> > From: "Victor Gene Hazlewood" <vhazlewo at utk.edu>
> > To: "Scott Jackson" <scottmo at adaptivecomputing.com>
> > Cc: "Gold Users Mailing List" <gold-users at supercluster.org>
> > Sent: Friday, November 19, 2010 8:42:43 AM
> > Subject: RE: [gold-users] Project doesn't get marked active
> > Well, maybe the project doesn’t get marked inactive. Does it? I see
> > the project got marked Inactive by gchproject command instead of
> > from
> > the Gold daemon. Gold definitely marks the allocation Active=True
> > and
> > Active=False correctly in these situations. Does it affect the
> > project? I’ll have to run a test to see.
> >
> >
> >
> > -Victor
> >
> >
> >
> >
> >
> > From: gold-users-bounces at supercluster.org
> > [mailto:gold-users-bounces at supercluster.org] On Behalf Of Hazlewood,
> > Victor Gene
> > Sent: Friday, November 19, 2010 10:33 AM
> > To: Scott Jackson
> > Cc: Gold Users Mailing List
> > Subject: [gold-users] Project doesn't get marked active
> >
> >
> >
> > Scott,
> >
> >
> >
> > Came across something interesting and wanted to know your thoughts.
> >
> >
> >
> > I am running Gold version 2.1.5.0 and if a project is marked
> > inactive
> > and a gchalloc changes the end date of an allocation whereby the
> > allocation (should) become active, Gold doesn't seem to change the
> > project state to active=True. I am wondering if this is by design or
> > perhaps a mistake. I want the behavior that if I change an
> > allocation
> > end date which now makes it active to have the project get marked
> > Active=True. Otherwise, why would Gold mark a project Inactive
> > automatically when the end date for an allocation passes? Seems like
> > this is the behavior that is desired. If the only allocation that is
> > active goes past its end date then Gold marks the project
> > Active=False. I am thinking if the end date of any allocation gets
> > changed to where an allocation is now active the project should be
> > set
> > Active=True.
> >
> >
> >
> > What do you think?
> >
> >
> >
> > The below illustrates my observation:
> >
> >
> >
> > -bash-3.2$ glsproject UT-NTNL0004
> >
> > Name Description Machines Active Users
> >
> > ----------- ----------------------------------- --------------------
> > ------ ---------------
> >
> > UT-NTNL0004 Modeling thunderstorms and tornados kraken.nics.teragrid
> > False user1,user2
> >
> >
> >
> > -bash-3.2$ glsaccount -a 40
> >
> > Id Name Amount Projects Users Machines Description
> >
> > -- ----------- ------ ----------- ---------------
> > --------------------
> > --------------
> >
> > 40 UT-NTNL0004 0.00 UT-NTNL0004 user1,user2 kraken.nics.teragrid
> > Auto-generated
> >
> >
> >
> > -bash-3.2$ glsalloc -a 40
> >
> > Id Account Active StartTime EndTime Amount CreditLimit Deposited
> > Description
> >
> > -- ------- ------ ---------- ------------------- ---------
> > -----------
> > --------- -----------
> >
> > 37 40 False 2008-08-01 2008-10-15 23:59:59 199989.68 0.00 200000.00
> >
> >
> >
> > -bash-3.2$ gchalloc -i 37 -e '2011-01-01'
> >
> > Successfully modified 1 Allocations
> >
> >
> >
> > -bash-3.2$ glsalloc -a 40
> >
> > Id Account Active StartTime EndTime Amount CreditLimit Deposited
> > Description
> >
> > -- ------- ------ ---------- ---------- --------- -----------
> > --------- -----------
> >
> > 37 40 True 2008-08-01 2011-01-01 199989.68 0.00 200000.00
> >
> >
> >
> > -bash-3.2$ glsaccount -a 40
> >
> > Id Name Amount Projects Users Machines Description
> >
> > -- ----------- --------- ----------- ---------------
> > -------------------- --------------
> >
> > 40 UT-NTNL0004 199989.68 UT-NTNL0004 user1,user2
> > kraken.nics.teragrid
> > Auto-generated
> >
> > -bash-3.2$ glsproject UT-NTNL0004
> >
> > Name Description Machines Active Users
> >
> > ----------- ----------------------------------- --------------------
> > ------ ---------------
> >
> > UT-NTNL0004 Modeling thunderstorms and tornados kraken.nics.teragrid
> > False user1,user2
> >
> >
> >
> > -Victor
> >
> >
> >
> > Victor Hazlewood, CISSP
> >
> > Senior HPC Systems Analyst
> >
> > National Institute for Computational Science
> >
> > University of Tennessee
> >
> > http://www.nics.tennessee.edu/


More information about the gold-users mailing list