[gold-users] importing old job data into gold

Hazlewood, Victor Gene vhazlewo at utk.edu
Wed Oct 6 07:52:22 MDT 2010


I don't know if this helps, but...

If all you are doing is wanting to have the job information in a table
and not use any of the other gold features (which I don't know if that
is what you want or not) why don't use just create a new Job table in
the Gold database by creating a new Gold Object with all the same
attributes as the Gold Job Object and then you could make your own load
program or modify the gmkjob to update your new Object instead of the
real one.

Or you could do it all with SQL commands of the database you are using
for Gold.   I have lots of extensions to Gold in our database both
created with Gold Objects and created by SQL commands.  I like creating
Gold Objects instead of using SQL commands because you get features like
when you load data it automagically gets creation times and
modifications times, etc.


-Victor

-----Original Message-----
From: gold-users-bounces at supercluster.org
[mailto:gold-users-bounces at supercluster.org] On Behalf Of Steve Crusan
Sent: Tuesday, October 05, 2010 4:39 PM
To: Gold Users Mailing List
Subject: Re: [gold-users] importing old job data into gold

Thanks for the response Scott. Gold is essentially a bank.

I guess I need to clarify my statement a bit more then. I don't care
about
applying old charges to people's balances, but more about starting at a
baseline with our old Maui logs. At this point I wouldn't care if after
running a massive script if every user/project/etc was millions of hours
in
debt. I just want the historical usage numbers, with a good way to query
them by time periods. Basically, my place would be to actually run gold
on
our cluster AFTER we do an import, change the balances, etc...

We're not at the point yet where we want to start using Gold as a
charging/CPU hours banking system, but more internally so we have easy
to
query numbers. I see an evolution coming soon, but let's not get ahead
of
ourselves.

Let's say I create a fresh instance of Gold, import any
users/groups/setting/etc, but do not have any job data. Now, just
importing
the Maui logs via a script would produce the behavior you specified
before,
(i.e job records get debited that day). Would there be any way of
changing
the times in the gold database itself to reflect the actual debit times?
I
guess it would be tricking the system if I changed all of the references
in
the right place, but would it be theoretically possible to change the
values
in the DB, so that some of the historical gold commands would reflect
the
actual job times?



Maybe I'm just trying to dig too deep into this, because it seems I can
use
a combination of glsjob + start/EndTimes to achieve what I'd like after
job
imports, but I'd rather not rewrite your gusage/statement/etc commands,
especially since they natively work with the db, and they will probably
be a
heck of a lot more efficient than my perl ;-)





On 10/1/10 6:44 PM, "Scott Jackson" <scottmo at adaptivecomputing.com>
wrote:

> Steve,
> 
> It was introduced in gold-2.2. If you are using 2.1 you will not have
gmkjob.
> 
> The imported jobs will not show up against the old statement, they
will show
> up against a current statement only. However, the jobs will be
inserted into
> the job table with the specified StartTime and EndTime (while
CreationTime and
> ModificationTime will be current dates). You have to realize that Gold
tracks
> the historical states of all allocations and objects and you can't
just insert
> debits into the past. This would disrupt and change every intervening
state.
> This would have effected balances causing other jobs not to have been
able to
> run, invalidating previously generated statements, etc. You would
never see a
> bank inserting credits or debits into their past balance sheets. If a
bank
> catches a mistake, they correct it in the present with a current
refund or
> withdrawal.
> 
> When you import the job data, it does not affect balances. If you wish
to
> catch up for the past charges, the charges will occur in the present.
> 
> Scott
> 
> 
> ----- Original Message -----
> From: "Steve Crusan" <scrusan at UR.Rochester.edu>
> To: gold-users at supercluster.org
> Sent: Friday, October 1, 2010 3:20:57 PM
> Subject: [gold-users] importing old job data into gold
> 
> 
> importing old job data into gold
> 
> Quick question about importing old job data (from Maui logs) into
Gold.
> 
> I've noticed that it seems to be possible to do this using the goldsh
shell
> (in the manual it shows gmkjob, but I cannot seem to find it; is it
> deprecated?): 
> 
> 
> goldsh Job Create JobId=PBS.$goldId.0 User=$netId Project=$account
> Machine=$machine Charge=$charge Processors=$procs StartTime=$startTime
> EndTime=$endTime WallDuration=$walltime
> 
>> From there, it seems one can use gcharge to charge the job:
> gcharge -p $account -u $netId -m $machine -P $procs -t $walltime -j
$goldId -d
> "charge imported from old data" -s $startTime -e $endTime
> 
> Now, provided that I properly fill in the correct attributes, and use
the
> proper gold job id, would doing the above operations properly keep the
> timestamps of the jobs? Meaning that if I have job logs from September
2009,
> if I import them and use their epoch time (do all the conversions,
etc), will
> the gstatement commands and other querying operations reflect the
proper
> dates/times? 
> 
> Ex: 
> gstatement -s 2009-09-10 -e 2009-12-25
> 
> Will the imported jobs from the Maui logs properly show up within that
> statement, and NOT show up for a rather recent query?
> 
> Thanks! 
> 
> 
> 
> ----------------------
> Steve Crusan 
> System Administrator
> Center for Research Computing
> 
> 
> _______________________________________________
> gold-users mailing list
> gold-users at supercluster.org
> http://www.supercluster.org/mailman/listinfo/gold-users
> _______________________________________________
> gold-users mailing list
> gold-users at supercluster.org
> http://www.supercluster.org/mailman/listinfo/gold-users



----------------------
Steve Crusan
System Administrator
Center for Research Computing
University of Rochester
https://www.crc.rochester.edu/

_______________________________________________
gold-users mailing list
gold-users at supercluster.org
http://www.supercluster.org/mailman/listinfo/gold-users


More information about the gold-users mailing list