[gold-users] importing old job data into gold

Steve Crusan scrusan at ur.rochester.edu
Wed Oct 6 17:49:49 MDT 2010


As we expand of usage of gold, I will definitely be adding some custom
objects, but that's after I straighten out the old data, and figure out a
better solution.

I think I am just going to use the gmkjob, but not charge them.

>From there I will write some sort of ghistory command that uses the Perl API
to query the gold job logs.

I think in the long run that will be better than trying to trick the
software, and fudge data in, especially given the situation Scott described
(I looked at some of those tables; yikes!). That will also essentially give
me a function for view historical usage, without having to worry about the
bank accounting part, especially since I have my suspicions that it will
become more important to us very soon.


Thanks for all the help!


On 10/6/10 9:52 AM, "Hazlewood, Victor Gene" <vhazlewo at utk.edu> wrote:

> I don't know if this helps, but...
> 
> If all you are doing is wanting to have the job information in a table
> and not use any of the other gold features (which I don't know if that
> is what you want or not) why don't use just create a new Job table in
> the Gold database by creating a new Gold Object with all the same
> attributes as the Gold Job Object and then you could make your own load
> program or modify the gmkjob to update your new Object instead of the
> real one.
> 
> Or you could do it all with SQL commands of the database you are using
> for Gold.   I have lots of extensions to Gold in our database both
> created with Gold Objects and created by SQL commands.  I like creating
> Gold Objects instead of using SQL commands because you get features like
> when you load data it automagically gets creation times and
> modifications times, etc.
> 
> 
> -Victor
> 
> -----Original Message-----
> From: gold-users-bounces at supercluster.org
> [mailto:gold-users-bounces at supercluster.org] On Behalf Of Steve Crusan
> Sent: Tuesday, October 05, 2010 4:39 PM
> To: Gold Users Mailing List
> Subject: Re: [gold-users] importing old job data into gold
> 
> Thanks for the response Scott. Gold is essentially a bank.
> 
> I guess I need to clarify my statement a bit more then. I don't care
> about
> applying old charges to people's balances, but more about starting at a
> baseline with our old Maui logs. At this point I wouldn't care if after
> running a massive script if every user/project/etc was millions of hours
> in
> debt. I just want the historical usage numbers, with a good way to query
> them by time periods. Basically, my place would be to actually run gold
> on
> our cluster AFTER we do an import, change the balances, etc...
> 
> We're not at the point yet where we want to start using Gold as a
> charging/CPU hours banking system, but more internally so we have easy
> to
> query numbers. I see an evolution coming soon, but let's not get ahead
> of
> ourselves.
> 
> Let's say I create a fresh instance of Gold, import any
> users/groups/setting/etc, but do not have any job data. Now, just
> importing
> the Maui logs via a script would produce the behavior you specified
> before,
> (i.e job records get debited that day). Would there be any way of
> changing
> the times in the gold database itself to reflect the actual debit times?
> I
> guess it would be tricking the system if I changed all of the references
> in
> the right place, but would it be theoretically possible to change the
> values
> in the DB, so that some of the historical gold commands would reflect
> the
> actual job times?
> 
> 
> 
> Maybe I'm just trying to dig too deep into this, because it seems I can
> use
> a combination of glsjob + start/EndTimes to achieve what I'd like after
> job
> imports, but I'd rather not rewrite your gusage/statement/etc commands,
> especially since they natively work with the db, and they will probably
> be a
> heck of a lot more efficient than my perl ;-)
> 
> 
> 
> 
> 
> On 10/1/10 6:44 PM, "Scott Jackson" <scottmo at adaptivecomputing.com>
> wrote:
> 
>> Steve,
>> 
>> It was introduced in gold-2.2. If you are using 2.1 you will not have
> gmkjob.
>> 
>> The imported jobs will not show up against the old statement, they
> will show
>> up against a current statement only. However, the jobs will be
> inserted into
>> the job table with the specified StartTime and EndTime (while
> CreationTime and
>> ModificationTime will be current dates). You have to realize that Gold
> tracks
>> the historical states of all allocations and objects and you can't
> just insert
>> debits into the past. This would disrupt and change every intervening
> state.
>> This would have effected balances causing other jobs not to have been
> able to
>> run, invalidating previously generated statements, etc. You would
> never see a
>> bank inserting credits or debits into their past balance sheets. If a
> bank
>> catches a mistake, they correct it in the present with a current
> refund or
>> withdrawal.
>> 
>> When you import the job data, it does not affect balances. If you wish
> to
>> catch up for the past charges, the charges will occur in the present.
>> 
>> Scott
>> 
>> 
>> ----- Original Message -----
>> From: "Steve Crusan" <scrusan at UR.Rochester.edu>
>> To: gold-users at supercluster.org
>> Sent: Friday, October 1, 2010 3:20:57 PM
>> Subject: [gold-users] importing old job data into gold
>> 
>> 
>> importing old job data into gold
>> 
>> Quick question about importing old job data (from Maui logs) into
> Gold.
>> 
>> I've noticed that it seems to be possible to do this using the goldsh
> shell
>> (in the manual it shows gmkjob, but I cannot seem to find it; is it
>> deprecated?): 
>> 
>> 
>> goldsh Job Create JobId=PBS.$goldId.0 User=$netId Project=$account
>> Machine=$machine Charge=$charge Processors=$procs StartTime=$startTime
>> EndTime=$endTime WallDuration=$walltime
>> 
>>> From there, it seems one can use gcharge to charge the job:
>> gcharge -p $account -u $netId -m $machine -P $procs -t $walltime -j
> $goldId -d
>> "charge imported from old data" -s $startTime -e $endTime
>> 
>> Now, provided that I properly fill in the correct attributes, and use
> the
>> proper gold job id, would doing the above operations properly keep the
>> timestamps of the jobs? Meaning that if I have job logs from September
> 2009,
>> if I import them and use their epoch time (do all the conversions,
> etc), will
>> the gstatement commands and other querying operations reflect the
> proper
>> dates/times? 
>> 
>> Ex: 
>> gstatement -s 2009-09-10 -e 2009-12-25
>> 
>> Will the imported jobs from the Maui logs properly show up within that
>> statement, and NOT show up for a rather recent query?
>> 
>> Thanks! 
>> 
>> 
>> 
>> ----------------------
>> Steve Crusan 
>> System Administrator
>> Center for Research Computing
>> 
>> 
>> _______________________________________________
>> gold-users mailing list
>> gold-users at supercluster.org
>> http://www.supercluster.org/mailman/listinfo/gold-users
>> _______________________________________________
>> gold-users mailing list
>> gold-users at supercluster.org
>> http://www.supercluster.org/mailman/listinfo/gold-users
> 
> 
> 
> ----------------------
> Steve Crusan
> System Administrator
> Center for Research Computing
> University of Rochester
> https://www.crc.rochester.edu/
> 
> _______________________________________________
> gold-users mailing list
> gold-users at supercluster.org
> http://www.supercluster.org/mailman/listinfo/gold-users
> _______________________________________________
> gold-users mailing list
> gold-users at supercluster.org
> http://www.supercluster.org/mailman/listinfo/gold-users



----------------------
Steve Crusan
System Administrator
Center for Research Computing
University of Rochester
https://www.crc.rochester.edu/



More information about the gold-users mailing list